2024-05-15T12:09:35.888 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-15T12:09:35.888 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_rzarzynski@teuthology --archive /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707093 --verbose --description rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} --name rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi -- /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707093/orig.config.yaml 2024-05-15T12:09:35.911 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-15T12:09:35.988 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707093 branch: wip-osd-ec-partial-reads description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} email: null first_in_suite: false job_id: '7707093' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-osd-ec-partial-reads ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 workunit: branch: wip-osd-ec-partial-reads sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 owner: scheduled_rzarzynski@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 8148 sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-osd-ec-partial-reads suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 targets: smithi005.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGZ7WMWDQYIDxvgoIH/K16opOd3KZ8H0EO+PrsAUOhJzL6a4sE9K4vrUjm6StTzGeSwQ1UHQ7MYtR/PhaTPkKQ8= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - erasure-code teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-15_11:53:37 tube: smithi user: rzarzynski verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-15T12:09:35.989 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa; will attempt to use it 2024-05-15T12:09:35.989 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks 2024-05-15T12:09:35.989 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-15T12:09:35.991 INFO:teuthology.task.internal:Checking packages... 2024-05-15T12:09:36.013 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'ae2481253f9b2fa3451954e615eedb67ebdd5314' 2024-05-15T12:09:36.013 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T12:09:36.014 INFO:teuthology.packaging:ref: None 2024-05-15T12:09:36.014 INFO:teuthology.packaging:tag: None 2024-05-15T12:09:36.014 INFO:teuthology.packaging:branch: wip-osd-ec-partial-reads 2024-05-15T12:09:36.014 INFO:teuthology.packaging:sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:09:36.014 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-osd-ec-partial-reads 2024-05-15T12:09:36.157 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1882-gae248125-1jammy 2024-05-15T12:09:36.159 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-15T12:09:36.164 INFO:teuthology.task.internal:no buildpackages task found 2024-05-15T12:09:36.164 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-15T12:09:36.169 INFO:teuthology.task.internal:Saving configuration 2024-05-15T12:09:36.179 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-15T12:09:36.184 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-15T12:09:36.208 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707093', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 12:03:56.773932', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII/1FpQXAAph7TRjMCMF9jVEAggAy7wIIZPWdKPa8yaq'} 2024-05-15T12:09:36.208 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-15T12:09:36.214 INFO:teuthology.task.internal:roles: ubuntu@smithi005.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-05-15T12:09:36.214 INFO:teuthology.run_tasks:Running task console_log... 2024-05-15T12:09:36.249 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc70b1541f0>, signals=[15]) 2024-05-15T12:09:36.250 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-15T12:09:36.255 INFO:teuthology.task.internal:Opening connections... 2024-05-15T12:09:36.255 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi005.front.sepia.ceph.com 2024-05-15T12:09:36.257 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:09:36.337 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-15T12:09:36.343 DEBUG:teuthology.orchestra.run.smithi005:> uname -m 2024-05-15T12:09:36.443 INFO:teuthology.orchestra.run.smithi005.stdout:x86_64 2024-05-15T12:09:36.443 DEBUG:teuthology.orchestra.run.smithi005:> cat /etc/os-release 2024-05-15T12:09:36.489 INFO:teuthology.orchestra.run.smithi005.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-05-15T12:09:36.489 INFO:teuthology.orchestra.run.smithi005.stdout:NAME="Ubuntu" 2024-05-15T12:09:36.489 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_ID="22.04" 2024-05-15T12:09:36.489 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-05-15T12:09:36.489 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_CODENAME=jammy 2024-05-15T12:09:36.490 INFO:teuthology.orchestra.run.smithi005.stdout:ID=ubuntu 2024-05-15T12:09:36.490 INFO:teuthology.orchestra.run.smithi005.stdout:ID_LIKE=debian 2024-05-15T12:09:36.490 INFO:teuthology.orchestra.run.smithi005.stdout:HOME_URL="https://www.ubuntu.com/" 2024-05-15T12:09:36.490 INFO:teuthology.orchestra.run.smithi005.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-05-15T12:09:36.490 INFO:teuthology.orchestra.run.smithi005.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-05-15T12:09:36.490 INFO:teuthology.orchestra.run.smithi005.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-05-15T12:09:36.490 INFO:teuthology.orchestra.run.smithi005.stdout:UBUNTU_CODENAME=jammy 2024-05-15T12:09:36.490 INFO:teuthology.lock.ops:Updating smithi005.front.sepia.ceph.com on lock server 2024-05-15T12:09:36.511 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-15T12:09:36.519 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-15T12:09:36.591 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-15T12:09:36.592 DEBUG:teuthology.orchestra.run.smithi005:> test '!' -e /home/ubuntu/cephtest 2024-05-15T12:09:36.596 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-15T12:09:36.602 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-15T12:09:36.603 DEBUG:teuthology.orchestra.run.smithi005:> test -z $(ls -A /var/lib/ceph) 2024-05-15T12:09:36.643 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-15T12:09:36.675 INFO:teuthology.run_tasks:Running task kernel... 2024-05-15T12:09:36.690 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-15T12:09:36.690 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-15T12:09:36.691 DEBUG:teuthology.orchestra.run.smithi005:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-15T12:09:36.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:09:36.696 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-05-15T12:09:36.742 INFO:teuthology.orchestra.run.smithi005.stdout:5.15.0-56-generic 2024-05-15T12:09:36.743 INFO:teuthology.task.kernel:Running kernel on smithi005: 5.15.0-56-generic 2024-05-15T12:09:36.743 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get clean 2024-05-15T12:09:36.858 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get update 2024-05-15T12:09:37.025 INFO:teuthology.orchestra.run.smithi005.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-05-15T12:09:37.108 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-15T12:09:37.190 INFO:teuthology.orchestra.run.smithi005.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-05-15T12:09:37.240 INFO:teuthology.orchestra.run.smithi005.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [469 kB] 2024-05-15T12:09:37.334 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,464 kB] 2024-05-15T12:09:37.394 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [252 kB] 2024-05-15T12:09:37.399 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-05-15T12:09:37.400 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-05-15T12:09:37.402 INFO:teuthology.orchestra.run.smithi005.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,863 kB] 2024-05-15T12:09:37.433 INFO:teuthology.orchestra.run.smithi005.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [316 kB] 2024-05-15T12:09:37.437 INFO:teuthology.orchestra.run.smithi005.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-05-15T12:09:37.437 INFO:teuthology.orchestra.run.smithi005.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [603 kB] 2024-05-15T12:09:37.443 INFO:teuthology.orchestra.run.smithi005.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-05-15T12:09:37.451 INFO:teuthology.orchestra.run.smithi005.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [164 kB] 2024-05-15T12:09:37.453 INFO:teuthology.orchestra.run.smithi005.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-05-15T12:09:37.453 INFO:teuthology.orchestra.run.smithi005.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-05-15T12:09:37.453 INFO:teuthology.orchestra.run.smithi005.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-05-15T12:09:37.454 INFO:teuthology.orchestra.run.smithi005.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-05-15T12:09:37.458 INFO:teuthology.orchestra.run.smithi005.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-05-15T12:09:37.554 INFO:teuthology.orchestra.run.smithi005.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-05-15T12:09:37.692 INFO:teuthology.orchestra.run.smithi005.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,678 kB] 2024-05-15T12:09:38.005 INFO:teuthology.orchestra.run.smithi005.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [631 kB] 2024-05-15T12:09:38.046 INFO:teuthology.orchestra.run.smithi005.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [311 kB] 2024-05-15T12:09:38.060 INFO:teuthology.orchestra.run.smithi005.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-05-15T12:09:38.061 INFO:teuthology.orchestra.run.smithi005.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-05-15T12:09:38.063 INFO:teuthology.orchestra.run.smithi005.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,923 kB] 2024-05-15T12:09:38.125 INFO:teuthology.orchestra.run.smithi005.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [327 kB] 2024-05-15T12:09:38.133 INFO:teuthology.orchestra.run.smithi005.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-05-15T12:09:38.297 INFO:teuthology.orchestra.run.smithi005.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [702 kB] 2024-05-15T12:09:38.886 INFO:teuthology.orchestra.run.smithi005.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,074 kB] 2024-05-15T12:09:38.998 INFO:teuthology.orchestra.run.smithi005.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [245 kB] 2024-05-15T12:09:39.018 INFO:teuthology.orchestra.run.smithi005.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-05-15T12:09:39.020 INFO:teuthology.orchestra.run.smithi005.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-05-15T12:09:39.020 INFO:teuthology.orchestra.run.smithi005.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-05-15T12:09:39.023 INFO:teuthology.orchestra.run.smithi005.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-05-15T12:09:39.024 INFO:teuthology.orchestra.run.smithi005.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-05-15T12:09:39.024 INFO:teuthology.orchestra.run.smithi005.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-05-15T12:09:39.028 INFO:teuthology.orchestra.run.smithi005.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-05-15T12:09:39.033 INFO:teuthology.orchestra.run.smithi005.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-05-15T12:09:39.199 INFO:teuthology.orchestra.run.smithi005.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-05-15T12:09:39.281 INFO:teuthology.orchestra.run.smithi005.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-05-15T12:09:39.389 INFO:teuthology.orchestra.run.smithi005.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-05-15T12:09:39.445 INFO:teuthology.orchestra.run.smithi005.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-05-15T12:09:39.476 INFO:teuthology.orchestra.run.smithi005.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-05-15T12:09:44.102 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 13.7 MB in 3s (5,347 kB/s) 2024-05-15T12:09:45.461 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-05-15T12:09:45.479 DEBUG:teuthology.orchestra.run.smithi005:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-05-15T12:09:45.540 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-05-15T12:09:45.745 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2024-05-15T12:09:45.746 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2024-05-15T12:09:45.934 INFO:teuthology.orchestra.run.smithi005.stdout:The following packages were automatically installed and are no longer required: 2024-05-15T12:09:45.934 INFO:teuthology.orchestra.run.smithi005.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-15T12:09:45.935 INFO:teuthology.orchestra.run.smithi005.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-15T12:09:45.935 INFO:teuthology.orchestra.run.smithi005.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-15T12:09:45.935 INFO:teuthology.orchestra.run.smithi005.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-05-15T12:09:45.935 INFO:teuthology.orchestra.run.smithi005.stdout: python2.7-minimal 2024-05-15T12:09:45.935 INFO:teuthology.orchestra.run.smithi005.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-15T12:09:45.935 INFO:teuthology.orchestra.run.smithi005.stdout:The following additional packages will be installed: 2024-05-15T12:09:45.935 INFO:teuthology.orchestra.run.smithi005.stdout: linux-generic linux-headers-5.15.0-107 linux-headers-5.15.0-107-generic 2024-05-15T12:09:45.935 INFO:teuthology.orchestra.run.smithi005.stdout: linux-headers-generic linux-image-5.15.0-107-generic 2024-05-15T12:09:45.936 INFO:teuthology.orchestra.run.smithi005.stdout: linux-modules-5.15.0-107-generic linux-modules-extra-5.15.0-107-generic 2024-05-15T12:09:45.937 INFO:teuthology.orchestra.run.smithi005.stdout:Suggested packages: 2024-05-15T12:09:45.937 INFO:teuthology.orchestra.run.smithi005.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-05-15T12:09:45.937 INFO:teuthology.orchestra.run.smithi005.stdout:Recommended packages: 2024-05-15T12:09:45.937 INFO:teuthology.orchestra.run.smithi005.stdout: thermald 2024-05-15T12:09:45.985 INFO:teuthology.orchestra.run.smithi005.stdout:The following NEW packages will be installed: 2024-05-15T12:09:45.985 INFO:teuthology.orchestra.run.smithi005.stdout: linux-headers-5.15.0-107 linux-headers-5.15.0-107-generic 2024-05-15T12:09:45.985 INFO:teuthology.orchestra.run.smithi005.stdout: linux-image-5.15.0-107-generic linux-modules-5.15.0-107-generic 2024-05-15T12:09:45.985 INFO:teuthology.orchestra.run.smithi005.stdout: linux-modules-extra-5.15.0-107-generic 2024-05-15T12:09:45.986 INFO:teuthology.orchestra.run.smithi005.stdout:The following packages will be upgraded: 2024-05-15T12:09:45.987 INFO:teuthology.orchestra.run.smithi005.stdout: linux-generic linux-headers-generic linux-image-generic 2024-05-15T12:09:46.044 INFO:teuthology.orchestra.run.smithi005.stdout:3 upgraded, 5 newly installed, 0 to remove and 320 not upgraded. 2024-05-15T12:09:46.044 INFO:teuthology.orchestra.run.smithi005.stdout:Need to get 113 MB of archives. 2024-05-15T12:09:46.044 INFO:teuthology.orchestra.run.smithi005.stdout:After this operation, 583 MB of additional disk space will be used. 2024-05-15T12:09:46.044 INFO:teuthology.orchestra.run.smithi005.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-107-generic amd64 5.15.0-107.117 [22.7 MB] 2024-05-15T12:09:46.545 INFO:teuthology.orchestra.run.smithi005.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-107-generic amd64 5.15.0-107.117 [11.5 MB] 2024-05-15T12:09:46.664 INFO:teuthology.orchestra.run.smithi005.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-107-generic amd64 5.15.0-107.117 [63.9 MB] 2024-05-15T12:09:47.601 INFO:teuthology.orchestra.run.smithi005.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.107.107 [1,694 B] 2024-05-15T12:09:47.602 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.107.107 [2,496 B] 2024-05-15T12:09:47.602 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-107 all 5.15.0-107.117 [12.3 MB] 2024-05-15T12:09:47.734 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-107-generic amd64 5.15.0-107.117 [2,887 kB] 2024-05-15T12:09:47.762 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.107.107 [2,354 B] 2024-05-15T12:09:48.112 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 113 MB in 2s (64.1 MB/s) 2024-05-15T12:09:48.268 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-modules-5.15.0-107-generic. 2024-05-15T12:09:50.251 INFO:teuthology.orchestra.run.smithi005.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-05-15T12:09:50.255 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../0-linux-modules-5.15.0-107-generic_5.15.0-107.117_amd64.deb ... 2024-05-15T12:09:50.341 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-modules-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-15T12:09:52.435 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-image-5.15.0-107-generic. 2024-05-15T12:09:52.457 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../1-linux-image-5.15.0-107-generic_5.15.0-107.117_amd64.deb ... 2024-05-15T12:09:52.555 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-image-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-15T12:09:53.029 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-107-generic. 2024-05-15T12:09:53.050 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-107-generic_5.15.0-107.117_amd64.deb ... 2024-05-15T12:09:53.088 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-modules-extra-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-15T12:09:59.805 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../3-linux-generic_5.15.0.107.107_amd64.deb ... 2024-05-15T12:09:59.908 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-generic (5.15.0.107.107) over (5.15.0.56.54) ... 2024-05-15T12:10:00.207 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.107.107_amd64.deb ... 2024-05-15T12:10:00.303 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-image-generic (5.15.0.107.107) over (5.15.0.56.54) ... 2024-05-15T12:10:00.631 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-headers-5.15.0-107. 2024-05-15T12:10:00.653 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../5-linux-headers-5.15.0-107_5.15.0-107.117_all.deb ... 2024-05-15T12:10:00.697 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-headers-5.15.0-107 (5.15.0-107.117) ... 2024-05-15T12:10:06.294 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-headers-5.15.0-107-generic. 2024-05-15T12:10:06.332 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../6-linux-headers-5.15.0-107-generic_5.15.0-107.117_amd64.deb ... 2024-05-15T12:10:06.369 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-headers-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-15T12:10:09.440 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.107.107_amd64.deb ... 2024-05-15T12:10:09.564 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-headers-generic (5.15.0.107.107) over (5.15.0.56.54) ... 2024-05-15T12:10:09.906 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-headers-5.15.0-107 (5.15.0-107.117) ... 2024-05-15T12:10:10.027 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-headers-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-15T12:10:10.153 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-headers-generic (5.15.0.107.107) ... 2024-05-15T12:10:10.283 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-image-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-15T12:10:11.622 INFO:teuthology.orchestra.run.smithi005.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-05-15T12:10:11.622 INFO:teuthology.orchestra.run.smithi005.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-05-15T12:10:11.622 INFO:teuthology.orchestra.run.smithi005.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-107-generic 2024-05-15T12:10:11.622 INFO:teuthology.orchestra.run.smithi005.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-107-generic 2024-05-15T12:10:11.785 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-modules-extra-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-15T12:10:13.619 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-modules-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-15T12:10:15.029 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-image-generic (5.15.0.107.107) ... 2024-05-15T12:10:15.138 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-generic (5.15.0.107.107) ... 2024-05-15T12:10:15.239 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for linux-image-5.15.0-107-generic (5.15.0-107.117) ... 2024-05-15T12:10:15.306 INFO:teuthology.orchestra.run.smithi005.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-05-15T12:10:15.306 INFO:teuthology.orchestra.run.smithi005.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-107-generic 2024-05-15T12:10:30.013 INFO:teuthology.orchestra.run.smithi005.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-05-15T12:10:30.013 INFO:teuthology.orchestra.run.smithi005.stdout:Sourcing file `/etc/default/grub' 2024-05-15T12:10:30.036 INFO:teuthology.orchestra.run.smithi005.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-15T12:10:30.059 INFO:teuthology.orchestra.run.smithi005.stdout:Generating grub configuration file ... 2024-05-15T12:10:30.318 INFO:teuthology.orchestra.run.smithi005.stdout:Found linux image: /boot/vmlinuz-5.15.0-107-generic 2024-05-15T12:10:30.341 INFO:teuthology.orchestra.run.smithi005.stdout:Found initrd image: /boot/initrd.img-5.15.0-107-generic 2024-05-15T12:10:30.617 INFO:teuthology.orchestra.run.smithi005.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-15T12:10:30.622 INFO:teuthology.orchestra.run.smithi005.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-15T12:10:30.689 INFO:teuthology.orchestra.run.smithi005.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-15T12:10:30.695 INFO:teuthology.orchestra.run.smithi005.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-15T12:10:30.859 INFO:teuthology.orchestra.run.smithi005.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-15T12:10:30.860 INFO:teuthology.orchestra.run.smithi005.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-05-15T12:10:30.860 INFO:teuthology.orchestra.run.smithi005.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-15T12:10:30.902 INFO:teuthology.orchestra.run.smithi005.stdout:done 2024-05-15T12:10:31.357 DEBUG:teuthology.orchestra.run.smithi005:> dpkg -s linux-image-generic 2024-05-15T12:10:31.383 INFO:teuthology.orchestra.run.smithi005.stdout:Package: linux-image-generic 2024-05-15T12:10:31.383 INFO:teuthology.orchestra.run.smithi005.stdout:Status: install ok installed 2024-05-15T12:10:31.384 INFO:teuthology.orchestra.run.smithi005.stdout:Priority: optional 2024-05-15T12:10:31.384 INFO:teuthology.orchestra.run.smithi005.stdout:Section: kernel 2024-05-15T12:10:31.384 INFO:teuthology.orchestra.run.smithi005.stdout:Installed-Size: 21 2024-05-15T12:10:31.384 INFO:teuthology.orchestra.run.smithi005.stdout:Maintainer: Ubuntu Kernel Team 2024-05-15T12:10:31.384 INFO:teuthology.orchestra.run.smithi005.stdout:Architecture: amd64 2024-05-15T12:10:31.384 INFO:teuthology.orchestra.run.smithi005.stdout:Source: linux-meta 2024-05-15T12:10:31.384 INFO:teuthology.orchestra.run.smithi005.stdout:Version: 5.15.0.107.107 2024-05-15T12:10:31.384 INFO:teuthology.orchestra.run.smithi005.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-107), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-05-15T12:10:31.384 INFO:teuthology.orchestra.run.smithi005.stdout:Depends: linux-image-5.15.0-107-generic, linux-modules-extra-5.15.0-107-generic, linux-firmware, intel-microcode, amd64-microcode 2024-05-15T12:10:31.384 INFO:teuthology.orchestra.run.smithi005.stdout:Recommends: thermald 2024-05-15T12:10:31.384 INFO:teuthology.orchestra.run.smithi005.stdout:Description: Generic Linux kernel image 2024-05-15T12:10:31.384 INFO:teuthology.orchestra.run.smithi005.stdout: This package will always depend on the latest generic kernel image 2024-05-15T12:10:31.384 INFO:teuthology.orchestra.run.smithi005.stdout: available. 2024-05-15T12:10:31.384 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-107-generic 2024-05-15T12:10:31.384 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-15T12:10:31.385 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-15T12:10:31.385 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-05-15T12:10:31.385 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi005.front.sepia.ceph.com, path=None, version=distro) 2024-05-15T12:10:31.385 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get clean 2024-05-15T12:10:31.689 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get update 2024-05-15T12:10:31.840 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-15T12:10:31.847 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-15T12:10:31.869 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-15T12:10:31.927 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-15T12:10:33.271 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-05-15T12:10:33.289 DEBUG:teuthology.orchestra.run.smithi005:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-05-15T12:10:33.350 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-05-15T12:10:33.560 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2024-05-15T12:10:33.560 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2024-05-15T12:10:33.750 INFO:teuthology.orchestra.run.smithi005.stdout:linux-image-generic is already the newest version (5.15.0.107.107). 2024-05-15T12:10:33.750 INFO:teuthology.orchestra.run.smithi005.stdout:The following packages were automatically installed and are no longer required: 2024-05-15T12:10:33.750 INFO:teuthology.orchestra.run.smithi005.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-15T12:10:33.750 INFO:teuthology.orchestra.run.smithi005.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-15T12:10:33.750 INFO:teuthology.orchestra.run.smithi005.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-15T12:10:33.751 INFO:teuthology.orchestra.run.smithi005.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-15T12:10:33.751 INFO:teuthology.orchestra.run.smithi005.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-15T12:10:33.751 INFO:teuthology.orchestra.run.smithi005.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-15T12:10:33.791 INFO:teuthology.orchestra.run.smithi005.stdout:0 upgraded, 0 newly installed, 0 to remove and 320 not upgraded. 2024-05-15T12:10:33.793 DEBUG:teuthology.orchestra.run.smithi005:> dpkg -s linux-image-generic 2024-05-15T12:10:33.809 INFO:teuthology.orchestra.run.smithi005.stdout:Package: linux-image-generic 2024-05-15T12:10:33.810 INFO:teuthology.orchestra.run.smithi005.stdout:Status: install ok installed 2024-05-15T12:10:33.810 INFO:teuthology.orchestra.run.smithi005.stdout:Priority: optional 2024-05-15T12:10:33.810 INFO:teuthology.orchestra.run.smithi005.stdout:Section: kernel 2024-05-15T12:10:33.810 INFO:teuthology.orchestra.run.smithi005.stdout:Installed-Size: 21 2024-05-15T12:10:33.810 INFO:teuthology.orchestra.run.smithi005.stdout:Maintainer: Ubuntu Kernel Team 2024-05-15T12:10:33.810 INFO:teuthology.orchestra.run.smithi005.stdout:Architecture: amd64 2024-05-15T12:10:33.810 INFO:teuthology.orchestra.run.smithi005.stdout:Source: linux-meta 2024-05-15T12:10:33.810 INFO:teuthology.orchestra.run.smithi005.stdout:Version: 5.15.0.107.107 2024-05-15T12:10:33.810 INFO:teuthology.orchestra.run.smithi005.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-107), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-05-15T12:10:33.810 INFO:teuthology.orchestra.run.smithi005.stdout:Depends: linux-image-5.15.0-107-generic, linux-modules-extra-5.15.0-107-generic, linux-firmware, intel-microcode, amd64-microcode 2024-05-15T12:10:33.810 INFO:teuthology.orchestra.run.smithi005.stdout:Recommends: thermald 2024-05-15T12:10:33.810 INFO:teuthology.orchestra.run.smithi005.stdout:Description: Generic Linux kernel image 2024-05-15T12:10:33.811 INFO:teuthology.orchestra.run.smithi005.stdout: This package will always depend on the latest generic kernel image 2024-05-15T12:10:33.811 INFO:teuthology.orchestra.run.smithi005.stdout: available. 2024-05-15T12:10:33.812 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-05-15T12:10:33.856 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.g6uVs6sr2q 2024-05-15T12:10:33.856 DEBUG:teuthology.orchestra.run.smithi005:> sudo cp /boot/grub/grub.cfg /tmp/tmp.g6uVs6sr2q 2024-05-15T12:10:33.913 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.g6uVs6sr2q 2024-05-15T12:10:34.033 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.g6uVs6sr2q is 10KB 2024-05-15T12:10:34.046 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.g6uVs6sr2q 2024-05-15T12:10:34.051 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-05-15T12:10:34.105 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-05-15T12:10:34.105 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-05-15T12:10:34.105 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-05-15T12:10:34.172 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-107-generic 2024-05-15T12:10:34.173 DEBUG:teuthology.orchestra.run.smithi005:> sudo update-grub 2024-05-15T12:10:35.400 INFO:teuthology.orchestra.run.smithi005.stderr:Sourcing file `/etc/default/grub' 2024-05-15T12:10:35.401 INFO:teuthology.orchestra.run.smithi005.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-15T12:10:35.413 INFO:teuthology.orchestra.run.smithi005.stderr:Generating grub configuration file ... 2024-05-15T12:10:35.586 INFO:teuthology.orchestra.run.smithi005.stderr:Found linux image: /boot/vmlinuz-5.15.0-107-generic 2024-05-15T12:10:35.593 INFO:teuthology.orchestra.run.smithi005.stderr:Found initrd image: /boot/initrd.img-5.15.0-107-generic 2024-05-15T12:10:35.805 INFO:teuthology.orchestra.run.smithi005.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-15T12:10:35.810 INFO:teuthology.orchestra.run.smithi005.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-15T12:10:35.855 INFO:teuthology.orchestra.run.smithi005.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-15T12:10:35.861 INFO:teuthology.orchestra.run.smithi005.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-15T12:10:35.994 INFO:teuthology.orchestra.run.smithi005.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-15T12:10:35.994 INFO:teuthology.orchestra.run.smithi005.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-05-15T12:10:35.994 INFO:teuthology.orchestra.run.smithi005.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-15T12:10:36.005 INFO:teuthology.orchestra.run.smithi005.stderr:done 2024-05-15T12:10:36.021 DEBUG:teuthology.orchestra.run.smithi005:> sudo shutdown -r now 2024-05-15T12:11:06.054 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-15T12:11:06.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-05-15T12:11:06.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:11:24.574 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.5 2024-05-15T12:11:33.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-05-15T12:11:33.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:12:33.642 DEBUG:teuthology.orchestra.remote:timed out 2024-05-15T12:12:45.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-05-15T12:12:45.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:12:45.943 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-05-15T12:12:46.532 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-05-15T12:12:46.532 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-107-generic"... 2024-05-15T12:12:46.533 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-05-15T12:12:46.577 INFO:teuthology.orchestra.run.smithi005.stdout:5.15.0-107-generic 2024-05-15T12:12:46.577 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-107-generic vs 5.15.0-107-generic 2024-05-15T12:12:46.577 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-15T12:12:46.577 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-05-15T12:12:47.578 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-15T12:12:47.578 DEBUG:teuthology.orchestra.run.smithi005:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-15T12:12:47.659 INFO:teuthology.orchestra.run.smithi005.stdout:ttyS1 2024-05-15T12:12:47.683 DEBUG:teuthology.parallel:result is None 2024-05-15T12:12:47.683 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-15T12:12:47.690 INFO:teuthology.task.internal:Creating test directory... 2024-05-15T12:12:47.690 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-15T12:12:47.695 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-15T12:12:47.703 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-15T12:12:47.708 INFO:teuthology.task.internal:Creating archive directory... 2024-05-15T12:12:47.708 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-15T12:12:47.754 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-15T12:12:47.761 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-15T12:12:47.761 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-15T12:12:47.820 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:12:47.831 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:12:47.832 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-15T12:12:47.839 INFO:teuthology.task.internal:Configuring sudo... 2024-05-15T12:12:47.839 DEBUG:teuthology.orchestra.run.smithi005:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-15T12:12:47.886 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-15T12:12:47.894 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-15T12:12:47.894 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-15T12:12:47.933 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T12:12:47.982 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T12:12:48.030 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-05-15T12:12:48.030 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-15T12:12:48.100 DEBUG:teuthology.orchestra.run.smithi005:> sudo service rsyslog restart 2024-05-15T12:12:48.172 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-15T12:12:48.179 INFO:teuthology.task.internal:Starting timer... 2024-05-15T12:12:48.179 INFO:teuthology.run_tasks:Running task pcp... 2024-05-15T12:12:48.188 INFO:teuthology.run_tasks:Running task selinux... 2024-05-15T12:12:48.195 DEBUG:teuthology.task.selinux:Excluding smithi005: OS 'ubuntu' does not support SELinux 2024-05-15T12:12:48.196 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-15T12:12:48.196 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-05-15T12:12:48.196 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-15T12:12:48.196 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-15T12:12:48.204 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-05-15T12:12:48.209 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-05-15T12:12:48.343 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-15T12:12:48.359 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-15T12:12:48.360 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi005.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-15T12:19:55.310 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi005.front.sepia.ceph.com')] 2024-05-15T12:19:55.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-05-15T12:19:55.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:19:55.393 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-05-15T12:19:55.466 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-05-15T12:19:55.466 INFO:teuthology.run_tasks:Running task clock... 2024-05-15T12:19:55.475 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-15T12:19:55.476 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-15T12:19:55.476 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T12:19:55.536 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:19:55 ntpd[17139]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-05-15T12:19:55.536 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:19:55 ntpd[17139]: Command line: ntpd -gq 2024-05-15T12:19:55.536 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:19:55 ntpd[17139]: ---------------------------------------------------- 2024-05-15T12:19:55.537 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:19:55 ntpd[17139]: ntp-4 is maintained by Network Time Foundation, 2024-05-15T12:19:55.537 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:19:55 ntpd[17139]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-05-15T12:19:55.537 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:19:55 ntpd[17139]: corporation. Support and training for ntp-4 are 2024-05-15T12:19:55.537 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:19:55 ntpd[17139]: available at https://www.nwtime.org/support 2024-05-15T12:19:55.537 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:19:55 ntpd[17139]: ---------------------------------------------------- 2024-05-15T12:19:55.537 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:19:55 ntpd[17139]: proto: precision = 0.046 usec (-24) 2024-05-15T12:19:55.537 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:19:55 ntpd[17139]: basedate set to 2022-02-04 2024-05-15T12:19:55.538 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:19:55 ntpd[17139]: gps base set to 2022-02-06 (week 2196) 2024-05-15T12:19:55.538 INFO:teuthology.orchestra.run.smithi005.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-15T12:19:55.538 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:19:55 ntpd[17139]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-15T12:19:55.538 INFO:teuthology.orchestra.run.smithi005.stderr:restrict ::: KOD does nothing without LIMITED. 2024-05-15T12:19:55.539 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:19:55 ntpd[17139]: restrict ::: KOD does nothing without LIMITED. 2024-05-15T12:19:55.540 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:19:55 ntpd[17139]: Listen and drop on 0 v6wildcard [::]:123 2024-05-15T12:19:55.540 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:19:55 ntpd[17139]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-05-15T12:19:55.540 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:19:55 ntpd[17139]: Listen normally on 2 lo 127.0.0.1:123 2024-05-15T12:19:55.540 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:19:55 ntpd[17139]: Listen normally on 3 ens1f0 172.21.15.5:123 2024-05-15T12:19:55.540 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:19:55 ntpd[17139]: Listen normally on 4 lo [::1]:123 2024-05-15T12:19:55.540 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:19:55 ntpd[17139]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1442%4]:123 2024-05-15T12:19:55.540 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:19:55 ntpd[17139]: Listening on routing socket on fd #22 for interface updates 2024-05-15T12:19:59.537 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:19:59 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:19:59.537 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:19:59 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:19:59.537 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:19:59 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:19:59.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:19:59 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:19:59.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:19:59 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:19:59.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:19:59 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:19:59.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:19:59 ntpd[17139]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-15T12:19:59.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:19:59 ntpd[17139]: can't open /var/log/ntpstats/peerstats.20240515: Permission denied 2024-05-15T12:20:00.537 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:00 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:20:00.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:00 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:20:00.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:00 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:20:00.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:00 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:20:00.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:00 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:20:00.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:00 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:20:00.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:00 ntpd[17139]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-15T12:20:00.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:00 ntpd[17139]: can't open /var/log/ntpstats/peerstats.20240515: Permission denied 2024-05-15T12:20:00.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:00 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:20:00.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:00 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:20:00.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:00 ntpd[17139]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-15T12:20:00.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:00 ntpd[17139]: can't open /var/log/ntpstats/peerstats.20240515: Permission denied 2024-05-15T12:20:01.537 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:01 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:20:01.537 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:01 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:20:01.537 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:01 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:20:01.537 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:01 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:20:01.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:01 ntpd[17139]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-15T12:20:01.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:01 ntpd[17139]: can't open /var/log/ntpstats/peerstats.20240515: Permission denied 2024-05-15T12:20:02.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:02 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:20:02.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:02 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:20:02.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:02 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:20:02.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:02 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:20:02.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:02 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:20:02.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:02 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:20:02.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:02 ntpd[17139]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-15T12:20:02.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:02 ntpd[17139]: can't open /var/log/ntpstats/peerstats.20240515: Permission denied 2024-05-15T12:20:02.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:02 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:20:02.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:02 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:20:02.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:02 ntpd[17139]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-15T12:20:02.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:02 ntpd[17139]: can't open /var/log/ntpstats/peerstats.20240515: Permission denied 2024-05-15T12:20:03.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:03 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:20:03.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:03 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:20:03.539 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:03 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:20:03.539 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:03 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:20:03.539 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:03 ntpd[17139]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-15T12:20:03.539 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:03 ntpd[17139]: can't open /var/log/ntpstats/peerstats.20240515: Permission denied 2024-05-15T12:20:04.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:04 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:20:04.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:04 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:20:04.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:04 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:20:04.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:04 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:20:04.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:04 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:20:04.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:04 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:20:04.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:04 ntpd[17139]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-15T12:20:04.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:04 ntpd[17139]: can't open /var/log/ntpstats/peerstats.20240515: Permission denied 2024-05-15T12:20:04.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:04 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:20:04.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:04 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:20:04.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:04 ntpd[17139]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-15T12:20:04.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:04 ntpd[17139]: can't open /var/log/ntpstats/peerstats.20240515: Permission denied 2024-05-15T12:20:05.537 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:05 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:20:05.537 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:05 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:20:05.537 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:05 ntpd[17139]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-15T12:20:05.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:05 ntpd[17139]: can't open /var/log/ntpstats/rawstats.20240515: Permission denied 2024-05-15T12:20:05.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:05 ntpd[17139]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-15T12:20:05.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:05 ntpd[17139]: can't open /var/log/ntpstats/peerstats.20240515: Permission denied 2024-05-15T12:20:05.538 INFO:teuthology.orchestra.run.smithi005.stdout:15 May 12:20:05 ntpd[17139]: ntpd: time slew -0.002997 s 2024-05-15T12:20:05.538 INFO:teuthology.orchestra.run.smithi005.stdout:ntpd: time slew -0.002997s 2024-05-15T12:20:05.538 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:05 ntpd[17139]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-05-15T12:20:05.539 INFO:teuthology.orchestra.run.smithi005.stderr:15 May 12:20:05 ntpd[17139]: can't open /var/log/ntpstats/loopstats.20240515: Permission denied 2024-05-15T12:20:05.588 INFO:teuthology.orchestra.run.smithi005.stdout: remote refid st t when poll reach delay offset jitter 2024-05-15T12:20:05.588 INFO:teuthology.orchestra.run.smithi005.stdout:============================================================================== 2024-05-15T12:20:05.588 INFO:teuthology.orchestra.run.smithi005.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-15T12:20:05.588 INFO:teuthology.orchestra.run.smithi005.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-15T12:20:05.588 INFO:teuthology.orchestra.run.smithi005.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-15T12:20:05.588 INFO:teuthology.orchestra.run.smithi005.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-15T12:20:05.589 INFO:teuthology.run_tasks:Running task install... 2024-05-15T12:20:05.597 DEBUG:teuthology.task.install:project ceph 2024-05-15T12:20:05.598 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314'}} 2024-05-15T12:20:05.598 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314'} 2024-05-15T12:20:05.598 INFO:teuthology.task.install:Using flavor: default 2024-05-15T12:20:05.603 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-15T12:20:05.603 INFO:teuthology.task.install:extra packages: [] 2024-05-15T12:20:05.603 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-key list | grep Ceph 2024-05-15T12:20:05.697 INFO:teuthology.orchestra.run.smithi005.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-05-15T12:20:05.727 INFO:teuthology.orchestra.run.smithi005.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-05-15T12:20:05.727 INFO:teuthology.orchestra.run.smithi005.stdout:uid [ unknown] Ceph.com (release key) 2024-05-15T12:20:05.728 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-05-15T12:20:05.729 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:20:05.950 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default/ 2024-05-15T12:20:05.950 INFO:teuthology.task.install.deb:Package version is 19.0.0-1882-gae248125-1jammy 2024-05-15T12:20:06.063 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-05-15T12:20:06.063 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-05-15T12:20:06.078 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get update 2024-05-15T12:20:06.305 INFO:teuthology.orchestra.run.smithi005.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy InRelease 2024-05-15T12:20:06.327 INFO:teuthology.orchestra.run.smithi005.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2024-05-15T12:20:06.351 INFO:teuthology.orchestra.run.smithi005.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy Release.gpg 2024-05-15T12:20:06.378 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-15T12:20:06.379 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-15T12:20:06.379 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,410 B] 2024-05-15T12:20:06.409 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-05-15T12:20:06.451 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-15T12:20:06.532 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-15T12:20:06.758 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 28.2 kB in 0s (57.2 kB/s) 2024-05-15T12:20:07.762 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-05-15T12:20:07.779 DEBUG:teuthology.orchestra.run.smithi005:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1882-gae248125-1jammy cephadm=19.0.0-1882-gae248125-1jammy ceph-mds=19.0.0-1882-gae248125-1jammy ceph-mgr=19.0.0-1882-gae248125-1jammy ceph-common=19.0.0-1882-gae248125-1jammy ceph-fuse=19.0.0-1882-gae248125-1jammy ceph-test=19.0.0-1882-gae248125-1jammy ceph-volume=19.0.0-1882-gae248125-1jammy radosgw=19.0.0-1882-gae248125-1jammy python3-rados=19.0.0-1882-gae248125-1jammy python3-rgw=19.0.0-1882-gae248125-1jammy python3-cephfs=19.0.0-1882-gae248125-1jammy python3-rbd=19.0.0-1882-gae248125-1jammy libcephfs2=19.0.0-1882-gae248125-1jammy libcephfs-dev=19.0.0-1882-gae248125-1jammy librados2=19.0.0-1882-gae248125-1jammy librbd1=19.0.0-1882-gae248125-1jammy rbd-fuse=19.0.0-1882-gae248125-1jammy 2024-05-15T12:20:07.835 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-05-15T12:20:08.017 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2024-05-15T12:20:08.017 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2024-05-15T12:20:08.199 INFO:teuthology.orchestra.run.smithi005.stdout:The following packages were automatically installed and are no longer required: 2024-05-15T12:20:08.199 INFO:teuthology.orchestra.run.smithi005.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-15T12:20:08.199 INFO:teuthology.orchestra.run.smithi005.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-15T12:20:08.199 INFO:teuthology.orchestra.run.smithi005.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-15T12:20:08.199 INFO:teuthology.orchestra.run.smithi005.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-15T12:20:08.199 INFO:teuthology.orchestra.run.smithi005.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-15T12:20:08.199 INFO:teuthology.orchestra.run.smithi005.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-15T12:20:08.201 INFO:teuthology.orchestra.run.smithi005.stdout:The following additional packages will be installed: 2024-05-15T12:20:08.201 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-05-15T12:20:08.201 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-05-15T12:20:08.201 INFO:teuthology.orchestra.run.smithi005.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-05-15T12:20:08.201 INFO:teuthology.orchestra.run.smithi005.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-05-15T12:20:08.201 INFO:teuthology.orchestra.run.smithi005.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-05-15T12:20:08.201 INFO:teuthology.orchestra.run.smithi005.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-05-15T12:20:08.201 INFO:teuthology.orchestra.run.smithi005.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-05-15T12:20:08.202 INFO:teuthology.orchestra.run.smithi005.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-05-15T12:20:08.202 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-05-15T12:20:08.202 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-05-15T12:20:08.202 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-05-15T12:20:08.202 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-05-15T12:20:08.202 INFO:teuthology.orchestra.run.smithi005.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-05-15T12:20:08.202 INFO:teuthology.orchestra.run.smithi005.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-05-15T12:20:08.202 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-05-15T12:20:08.202 INFO:teuthology.orchestra.run.smithi005.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-05-15T12:20:08.202 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-05-15T12:20:08.202 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-05-15T12:20:08.202 INFO:teuthology.orchestra.run.smithi005.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-05-15T12:20:08.202 INFO:teuthology.orchestra.run.smithi005.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-05-15T12:20:08.203 INFO:teuthology.orchestra.run.smithi005.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-05-15T12:20:08.203 INFO:teuthology.orchestra.run.smithi005.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-05-15T12:20:08.203 INFO:teuthology.orchestra.run.smithi005.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-05-15T12:20:08.203 INFO:teuthology.orchestra.run.smithi005.stdout: xmlstarlet zip 2024-05-15T12:20:08.203 INFO:teuthology.orchestra.run.smithi005.stdout:Suggested packages: 2024-05-15T12:20:08.203 INFO:teuthology.orchestra.run.smithi005.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-05-15T12:20:08.203 INFO:teuthology.orchestra.run.smithi005.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-05-15T12:20:08.203 INFO:teuthology.orchestra.run.smithi005.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-05-15T12:20:08.203 INFO:teuthology.orchestra.run.smithi005.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-05-15T12:20:08.203 INFO:teuthology.orchestra.run.smithi005.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-05-15T12:20:08.203 INFO:teuthology.orchestra.run.smithi005.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-05-15T12:20:08.204 INFO:teuthology.orchestra.run.smithi005.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-05-15T12:20:08.204 INFO:teuthology.orchestra.run.smithi005.stdout:Recommended packages: 2024-05-15T12:20:08.204 INFO:teuthology.orchestra.run.smithi005.stdout: btrfs-tools 2024-05-15T12:20:08.306 INFO:teuthology.orchestra.run.smithi005.stdout:The following NEW packages will be installed: 2024-05-15T12:20:08.306 INFO:teuthology.orchestra.run.smithi005.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-05-15T12:20:08.306 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-05-15T12:20:08.306 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-05-15T12:20:08.306 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-05-15T12:20:08.306 INFO:teuthology.orchestra.run.smithi005.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-05-15T12:20:08.307 INFO:teuthology.orchestra.run.smithi005.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-05-15T12:20:08.308 INFO:teuthology.orchestra.run.smithi005.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-05-15T12:20:08.308 INFO:teuthology.orchestra.run.smithi005.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-05-15T12:20:08.308 INFO:teuthology.orchestra.run.smithi005.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-05-15T12:20:08.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-05-15T12:20:08.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-05-15T12:20:08.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-05-15T12:20:08.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-05-15T12:20:08.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-05-15T12:20:08.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-05-15T12:20:08.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-05-15T12:20:08.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-05-15T12:20:08.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-05-15T12:20:08.309 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-05-15T12:20:08.309 INFO:teuthology.orchestra.run.smithi005.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-05-15T12:20:08.309 INFO:teuthology.orchestra.run.smithi005.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-05-15T12:20:08.309 INFO:teuthology.orchestra.run.smithi005.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-05-15T12:20:08.309 INFO:teuthology.orchestra.run.smithi005.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-05-15T12:20:08.309 INFO:teuthology.orchestra.run.smithi005.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-05-15T12:20:08.309 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-05-15T12:20:08.309 INFO:teuthology.orchestra.run.smithi005.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-05-15T12:20:08.496 INFO:teuthology.orchestra.run.smithi005.stdout:0 upgraded, 118 newly installed, 0 to remove and 307 not upgraded. 2024-05-15T12:20:08.496 INFO:teuthology.orchestra.run.smithi005.stdout:Need to get 176 MB of archives. 2024-05-15T12:20:08.496 INFO:teuthology.orchestra.run.smithi005.stdout:After this operation, 730 MB of additional disk space will be used. 2024-05-15T12:20:08.497 INFO:teuthology.orchestra.run.smithi005.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-05-15T12:20:08.532 INFO:teuthology.orchestra.run.smithi005.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1882-gae248125-1jammy [3,544 kB] 2024-05-15T12:20:08.889 INFO:teuthology.orchestra.run.smithi005.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1882-gae248125-1jammy [3,220 kB] 2024-05-15T12:20:08.935 INFO:teuthology.orchestra.run.smithi005.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-05-15T12:20:08.947 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-05-15T12:20:08.990 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1882-gae248125-1jammy [711 kB] 2024-05-15T12:20:09.013 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1882-gae248125-1jammy [356 kB] 2024-05-15T12:20:09.025 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1882-gae248125-1jammy [32.8 kB] 2024-05-15T12:20:09.026 INFO:teuthology.orchestra.run.smithi005.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1882-gae248125-1jammy [180 kB] 2024-05-15T12:20:09.029 INFO:teuthology.orchestra.run.smithi005.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-05-15T12:20:09.031 INFO:teuthology.orchestra.run.smithi005.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1882-gae248125-1jammy [64.5 kB] 2024-05-15T12:20:09.033 INFO:teuthology.orchestra.run.smithi005.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1882-gae248125-1jammy [330 kB] 2024-05-15T12:20:09.044 INFO:teuthology.orchestra.run.smithi005.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1882-gae248125-1jammy [6,575 kB] 2024-05-15T12:20:09.261 INFO:teuthology.orchestra.run.smithi005.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1882-gae248125-1jammy [112 kB] 2024-05-15T12:20:09.264 INFO:teuthology.orchestra.run.smithi005.stdout:Get:15 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1882-gae248125-1jammy [467 kB] 2024-05-15T12:20:09.266 INFO:teuthology.orchestra.run.smithi005.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-05-15T12:20:09.276 INFO:teuthology.orchestra.run.smithi005.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-05-15T12:20:09.280 INFO:teuthology.orchestra.run.smithi005.stdout:Get:18 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1882-gae248125-1jammy [24.2 MB] 2024-05-15T12:20:09.308 INFO:teuthology.orchestra.run.smithi005.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-05-15T12:20:09.321 INFO:teuthology.orchestra.run.smithi005.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-05-15T12:20:09.322 INFO:teuthology.orchestra.run.smithi005.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-05-15T12:20:09.323 INFO:teuthology.orchestra.run.smithi005.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-05-15T12:20:09.340 INFO:teuthology.orchestra.run.smithi005.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-05-15T12:20:09.510 INFO:teuthology.orchestra.run.smithi005.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-05-15T12:20:09.918 INFO:teuthology.orchestra.run.smithi005.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-05-15T12:20:09.953 INFO:teuthology.orchestra.run.smithi005.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-05-15T12:20:09.955 INFO:teuthology.orchestra.run.smithi005.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-05-15T12:20:10.034 INFO:teuthology.orchestra.run.smithi005.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-05-15T12:20:10.086 INFO:teuthology.orchestra.run.smithi005.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-05-15T12:20:10.108 INFO:teuthology.orchestra.run.smithi005.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-05-15T12:20:10.114 INFO:teuthology.orchestra.run.smithi005.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-05-15T12:20:10.115 INFO:teuthology.orchestra.run.smithi005.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-05-15T12:20:10.126 INFO:teuthology.orchestra.run.smithi005.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-05-15T12:20:10.128 INFO:teuthology.orchestra.run.smithi005.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-05-15T12:20:10.242 INFO:teuthology.orchestra.run.smithi005.stdout:Get:35 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1882-gae248125-1jammy [5,081 kB] 2024-05-15T12:20:10.294 INFO:teuthology.orchestra.run.smithi005.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-05-15T12:20:10.377 INFO:teuthology.orchestra.run.smithi005.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-05-15T12:20:10.383 INFO:teuthology.orchestra.run.smithi005.stdout:Get:38 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1882-gae248125-1jammy [243 kB] 2024-05-15T12:20:10.388 INFO:teuthology.orchestra.run.smithi005.stdout:Get:39 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1882-gae248125-1jammy [124 kB] 2024-05-15T12:20:10.406 INFO:teuthology.orchestra.run.smithi005.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-05-15T12:20:10.423 INFO:teuthology.orchestra.run.smithi005.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-05-15T12:20:10.467 INFO:teuthology.orchestra.run.smithi005.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-05-15T12:20:10.628 INFO:teuthology.orchestra.run.smithi005.stdout:Get:43 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1882-gae248125-1jammy [1,520 kB] 2024-05-15T12:20:10.683 INFO:teuthology.orchestra.run.smithi005.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-05-15T12:20:10.698 INFO:teuthology.orchestra.run.smithi005.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-05-15T12:20:10.705 INFO:teuthology.orchestra.run.smithi005.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-05-15T12:20:10.710 INFO:teuthology.orchestra.run.smithi005.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-05-15T12:20:10.731 INFO:teuthology.orchestra.run.smithi005.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-05-15T12:20:10.735 INFO:teuthology.orchestra.run.smithi005.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-05-15T12:20:10.825 INFO:teuthology.orchestra.run.smithi005.stdout:Get:50 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1882-gae248125-1jammy [6,248 kB] 2024-05-15T12:20:10.902 INFO:teuthology.orchestra.run.smithi005.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-05-15T12:20:11.141 INFO:teuthology.orchestra.run.smithi005.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1882-gae248125-1jammy [22.8 MB] 2024-05-15T12:20:11.233 INFO:teuthology.orchestra.run.smithi005.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-05-15T12:20:11.271 INFO:teuthology.orchestra.run.smithi005.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-05-15T12:20:11.284 INFO:teuthology.orchestra.run.smithi005.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-05-15T12:20:11.496 INFO:teuthology.orchestra.run.smithi005.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-05-15T12:20:11.497 INFO:teuthology.orchestra.run.smithi005.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-05-15T12:20:11.502 INFO:teuthology.orchestra.run.smithi005.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-05-15T12:20:11.508 INFO:teuthology.orchestra.run.smithi005.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-05-15T12:20:11.522 INFO:teuthology.orchestra.run.smithi005.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-05-15T12:20:11.544 INFO:teuthology.orchestra.run.smithi005.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-05-15T12:20:11.555 INFO:teuthology.orchestra.run.smithi005.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-05-15T12:20:11.735 INFO:teuthology.orchestra.run.smithi005.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-05-15T12:20:12.000 INFO:teuthology.orchestra.run.smithi005.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-05-15T12:20:12.255 INFO:teuthology.orchestra.run.smithi005.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-05-15T12:20:12.256 INFO:teuthology.orchestra.run.smithi005.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-05-15T12:20:12.256 INFO:teuthology.orchestra.run.smithi005.stdout:Get:67 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1882-gae248125-1jammy [14.0 kB] 2024-05-15T12:20:12.257 INFO:teuthology.orchestra.run.smithi005.stdout:Get:68 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1882-gae248125-1jammy [867 kB] 2024-05-15T12:20:12.257 INFO:teuthology.orchestra.run.smithi005.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-05-15T12:20:12.275 INFO:teuthology.orchestra.run.smithi005.stdout:Get:70 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1882-gae248125-1jammy [2,192 kB] 2024-05-15T12:20:12.276 INFO:teuthology.orchestra.run.smithi005.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-05-15T12:20:12.314 INFO:teuthology.orchestra.run.smithi005.stdout:Get:72 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1882-gae248125-1jammy [777 kB] 2024-05-15T12:20:12.324 INFO:teuthology.orchestra.run.smithi005.stdout:Get:73 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1882-gae248125-1jammy [144 kB] 2024-05-15T12:20:12.326 INFO:teuthology.orchestra.run.smithi005.stdout:Get:74 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1882-gae248125-1jammy [1,923 kB] 2024-05-15T12:20:12.416 INFO:teuthology.orchestra.run.smithi005.stdout:Get:75 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1882-gae248125-1jammy [8,625 kB] 2024-05-15T12:20:12.596 INFO:teuthology.orchestra.run.smithi005.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-05-15T12:20:12.598 INFO:teuthology.orchestra.run.smithi005.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-05-15T12:20:12.601 INFO:teuthology.orchestra.run.smithi005.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-05-15T12:20:12.602 INFO:teuthology.orchestra.run.smithi005.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-05-15T12:20:12.604 INFO:teuthology.orchestra.run.smithi005.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-05-15T12:20:12.788 INFO:teuthology.orchestra.run.smithi005.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1882-gae248125-1jammy [14.2 kB] 2024-05-15T12:20:12.788 INFO:teuthology.orchestra.run.smithi005.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1882-gae248125-1jammy [49.9 MB] 2024-05-15T12:20:12.833 INFO:teuthology.orchestra.run.smithi005.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-05-15T12:20:13.269 INFO:teuthology.orchestra.run.smithi005.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-05-15T12:20:13.275 INFO:teuthology.orchestra.run.smithi005.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-05-15T12:20:13.566 INFO:teuthology.orchestra.run.smithi005.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-05-15T12:20:13.566 INFO:teuthology.orchestra.run.smithi005.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-05-15T12:20:13.568 INFO:teuthology.orchestra.run.smithi005.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-05-15T12:20:13.570 INFO:teuthology.orchestra.run.smithi005.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-05-15T12:20:13.572 INFO:teuthology.orchestra.run.smithi005.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-05-15T12:20:13.574 INFO:teuthology.orchestra.run.smithi005.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-05-15T12:20:13.575 INFO:teuthology.orchestra.run.smithi005.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-05-15T12:20:13.577 INFO:teuthology.orchestra.run.smithi005.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-05-15T12:20:13.758 INFO:teuthology.orchestra.run.smithi005.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-05-15T12:20:14.166 INFO:teuthology.orchestra.run.smithi005.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-05-15T12:20:14.213 INFO:teuthology.orchestra.run.smithi005.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-05-15T12:20:14.250 INFO:teuthology.orchestra.run.smithi005.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-05-15T12:20:14.356 INFO:teuthology.orchestra.run.smithi005.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-05-15T12:20:14.397 INFO:teuthology.orchestra.run.smithi005.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-05-15T12:20:14.409 INFO:teuthology.orchestra.run.smithi005.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-05-15T12:20:14.415 INFO:teuthology.orchestra.run.smithi005.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-05-15T12:20:14.462 INFO:teuthology.orchestra.run.smithi005.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-05-15T12:20:14.467 INFO:teuthology.orchestra.run.smithi005.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-05-15T12:20:14.487 INFO:teuthology.orchestra.run.smithi005.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-05-15T12:20:14.652 INFO:teuthology.orchestra.run.smithi005.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-05-15T12:20:14.925 INFO:teuthology.orchestra.run.smithi005.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-05-15T12:20:14.942 INFO:teuthology.orchestra.run.smithi005.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-05-15T12:20:15.096 INFO:teuthology.orchestra.run.smithi005.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-05-15T12:20:15.125 INFO:teuthology.orchestra.run.smithi005.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-05-15T12:20:15.347 INFO:teuthology.orchestra.run.smithi005.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-05-15T12:20:15.349 INFO:teuthology.orchestra.run.smithi005.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-05-15T12:20:15.350 INFO:teuthology.orchestra.run.smithi005.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-05-15T12:20:15.369 INFO:teuthology.orchestra.run.smithi005.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-05-15T12:20:15.372 INFO:teuthology.orchestra.run.smithi005.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-05-15T12:20:15.567 INFO:teuthology.orchestra.run.smithi005.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1882-gae248125-1jammy [121 kB] 2024-05-15T12:20:15.569 INFO:teuthology.orchestra.run.smithi005.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1882-gae248125-1jammy [38.4 kB] 2024-05-15T12:20:15.570 INFO:teuthology.orchestra.run.smithi005.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1882-gae248125-1jammy [13.0 MB] 2024-05-15T12:20:15.956 INFO:teuthology.orchestra.run.smithi005.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1882-gae248125-1jammy [91.3 kB] 2024-05-15T12:20:16.554 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 176 MB in 8s (23.0 MB/s) 2024-05-15T12:20:16.720 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-05-15T12:20:16.823 INFO:teuthology.orchestra.run.smithi005.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-05-15T12:20:16.828 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-05-15T12:20:16.927 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-15T12:20:17.272 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-05-15T12:20:17.299 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-05-15T12:20:17.338 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-15T12:20:17.683 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-05-15T12:20:17.710 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-05-15T12:20:17.750 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-15T12:20:18.178 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-05-15T12:20:18.205 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-15T12:20:18.253 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-15T12:20:18.631 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-05-15T12:20:18.648 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-15T12:20:18.681 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-15T12:20:19.026 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-05-15T12:20:19.053 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-15T12:20:19.092 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-15T12:20:19.470 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-05-15T12:20:19.497 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-05-15T12:20:19.537 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-15T12:20:19.848 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package librados2. 2024-05-15T12:20:19.875 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../007-librados2_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:20:19.906 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librados2 (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:20.351 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package librbd1. 2024-05-15T12:20:20.377 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../008-librbd1_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:20:20.417 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librbd1 (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:20.878 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libcephfs2. 2024-05-15T12:20:20.905 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:20:20.945 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libcephfs2 (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:21.231 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rados. 2024-05-15T12:20:21.257 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../010-python3-rados_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:20:21.297 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rados (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:21.592 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-05-15T12:20:21.618 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-1882-gae248125-1jammy_all.deb ... 2024-05-15T12:20:21.650 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-ceph-argparse (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:21.885 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cephfs. 2024-05-15T12:20:21.902 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:20:21.944 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cephfs (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:22.230 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-ceph-common. 2024-05-15T12:20:22.257 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-1882-gae248125-1jammy_all.deb ... 2024-05-15T12:20:22.296 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-ceph-common (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:22.593 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-wcwidth. 2024-05-15T12:20:22.618 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-05-15T12:20:22.649 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-15T12:20:22.952 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-prettytable. 2024-05-15T12:20:22.979 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-05-15T12:20:23.019 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-05-15T12:20:23.279 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rbd. 2024-05-15T12:20:23.295 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:20:23.338 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rbd (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:23.683 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-05-15T12:20:23.709 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-05-15T12:20:23.749 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-15T12:20:24.018 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-05-15T12:20:24.045 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-05-15T12:20:24.285 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-05-15T12:20:24.630 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-05-15T12:20:24.656 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-05-15T12:20:24.696 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-15T12:20:25.041 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package lua5.1. 2024-05-15T12:20:25.068 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-05-15T12:20:25.108 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-05-15T12:20:25.402 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package lua-any. 2024-05-15T12:20:25.429 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-05-15T12:20:25.460 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking lua-any (27ubuntu1) ... 2024-05-15T12:20:25.680 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package zip. 2024-05-15T12:20:25.706 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-05-15T12:20:25.746 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking zip (3.0-12build2) ... 2024-05-15T12:20:26.100 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package unzip. 2024-05-15T12:20:26.126 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-05-15T12:20:26.166 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-05-15T12:20:26.578 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package luarocks. 2024-05-15T12:20:26.605 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-05-15T12:20:26.645 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-05-15T12:20:27.061 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package librgw2. 2024-05-15T12:20:27.088 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../025-librgw2_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:20:27.135 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librgw2 (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:27.730 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rgw. 2024-05-15T12:20:27.757 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:20:27.804 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rgw (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:28.158 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liboath0:amd64. 2024-05-15T12:20:28.185 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-05-15T12:20:28.224 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-05-15T12:20:28.560 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libradosstriper1. 2024-05-15T12:20:28.587 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:20:28.627 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libradosstriper1 (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:29.164 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-common. 2024-05-15T12:20:29.190 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../029-ceph-common_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:20:29.238 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-common (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:30.694 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-base. 2024-05-15T12:20:30.721 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../030-ceph-base_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:20:30.778 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-base (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:31.365 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-05-15T12:20:31.392 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-05-15T12:20:31.431 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-05-15T12:20:31.734 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cheroot. 2024-05-15T12:20:31.761 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-05-15T12:20:31.800 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-15T12:20:32.128 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-05-15T12:20:32.156 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-05-15T12:20:32.195 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-05-15T12:20:32.514 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-jaraco.text. 2024-05-15T12:20:32.541 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-05-15T12:20:32.572 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-05-15T12:20:32.867 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-05-15T12:20:32.894 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-05-15T12:20:32.934 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-05-15T12:20:33.236 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-tempora. 2024-05-15T12:20:33.264 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-05-15T12:20:33.303 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-05-15T12:20:33.581 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-portend. 2024-05-15T12:20:33.608 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-05-15T12:20:33.647 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-05-15T12:20:33.917 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-05-15T12:20:33.944 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-05-15T12:20:33.983 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-05-15T12:20:34.287 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cherrypy3. 2024-05-15T12:20:34.313 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-05-15T12:20:34.353 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-05-15T12:20:34.657 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-natsort. 2024-05-15T12:20:34.684 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-05-15T12:20:34.715 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-05-15T12:20:35.026 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-logutils. 2024-05-15T12:20:35.053 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-05-15T12:20:35.109 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-05-15T12:20:35.538 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-markupsafe. 2024-05-15T12:20:35.565 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-05-15T12:20:35.604 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-05-15T12:20:35.907 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-mako. 2024-05-15T12:20:35.934 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-05-15T12:20:35.982 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-15T12:20:36.369 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-simplegeneric. 2024-05-15T12:20:36.395 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-05-15T12:20:36.443 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-05-15T12:20:36.813 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-singledispatch. 2024-05-15T12:20:36.840 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-05-15T12:20:36.887 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-05-15T12:20:37.224 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-webob. 2024-05-15T12:20:37.251 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-05-15T12:20:37.290 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-05-15T12:20:37.635 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-waitress. 2024-05-15T12:20:37.662 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-05-15T12:20:37.705 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-15T12:20:38.021 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-tempita. 2024-05-15T12:20:38.049 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-05-15T12:20:38.088 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-05-15T12:20:38.399 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-paste. 2024-05-15T12:20:38.426 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-05-15T12:20:38.465 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-05-15T12:20:38.777 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-05-15T12:20:38.793 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-05-15T12:20:38.835 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-05-15T12:20:39.305 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pastedeploy. 2024-05-15T12:20:39.331 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-05-15T12:20:39.379 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-05-15T12:20:39.991 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-webtest. 2024-05-15T12:20:40.018 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-05-15T12:20:40.066 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-05-15T12:20:40.411 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pecan. 2024-05-15T12:20:40.437 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-05-15T12:20:40.552 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-05-15T12:20:40.898 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-certifi. 2024-05-15T12:20:40.925 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-05-15T12:20:40.964 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-05-15T12:20:41.276 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-idna. 2024-05-15T12:20:41.292 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-05-15T12:20:41.334 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-idna (3.3-1) ... 2024-05-15T12:20:42.488 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-urllib3. 2024-05-15T12:20:42.515 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-05-15T12:20:42.554 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-15T12:20:42.866 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-requests. 2024-05-15T12:20:42.893 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-05-15T12:20:42.932 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-15T12:20:43.244 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-werkzeug. 2024-05-15T12:20:43.271 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-05-15T12:20:43.310 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-15T12:20:43.580 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-05-15T12:20:43.596 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-1882-gae248125-1jammy_all.deb ... 2024-05-15T12:20:43.638 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:44.043 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-05-15T12:20:44.070 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:20:44.109 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:44.362 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-bcrypt. 2024-05-15T12:20:44.389 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-05-15T12:20:44.420 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-05-15T12:20:44.723 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-openssl. 2024-05-15T12:20:44.750 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-05-15T12:20:44.789 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-05-15T12:20:45.176 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr. 2024-05-15T12:20:45.203 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:20:45.242 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:45.604 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mon. 2024-05-15T12:20:45.631 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:20:45.662 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mon (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:46.241 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-osd. 2024-05-15T12:20:46.268 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:20:46.315 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-osd (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:47.485 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph. 2024-05-15T12:20:47.512 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../066-ceph_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:20:47.552 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:47.846 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-fuse. 2024-05-15T12:20:47.873 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:20:47.912 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-fuse (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:48.232 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mds. 2024-05-15T12:20:48.248 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:20:48.292 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mds (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:48.696 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package cephadm. 2024-05-15T12:20:48.722 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../069-cephadm_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:20:48.762 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking cephadm (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:49.048 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-asyncssh. 2024-05-15T12:20:49.064 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-05-15T12:20:49.097 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-05-15T12:20:49.434 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python-babel-localedata. 2024-05-15T12:20:49.461 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-05-15T12:20:49.500 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-15T12:20:50.722 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-babel. 2024-05-15T12:20:50.738 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-05-15T12:20:50.771 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-05-15T12:20:51.066 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-jinja2. 2024-05-15T12:20:51.094 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-05-15T12:20:51.133 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-15T12:20:51.494 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-05-15T12:20:51.521 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-1882-gae248125-1jammy_all.deb ... 2024-05-15T12:20:51.561 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:51.881 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-repoze.lru. 2024-05-15T12:20:51.908 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-05-15T12:20:51.947 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-05-15T12:20:52.250 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-routes. 2024-05-15T12:20:52.277 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-05-15T12:20:52.316 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-05-15T12:20:52.661 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-05-15T12:20:52.689 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-1882-gae248125-1jammy_all.deb ... 2024-05-15T12:20:52.727 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:53.164 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-05-15T12:20:53.180 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-05-15T12:20:53.222 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-15T12:20:53.693 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-joblib. 2024-05-15T12:20:53.720 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-05-15T12:20:53.750 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-05-15T12:20:54.121 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-05-15T12:20:54.148 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-05-15T12:20:54.187 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-05-15T12:20:54.498 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-sklearn. 2024-05-15T12:20:54.525 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-05-15T12:20:54.564 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-15T12:20:55.256 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-05-15T12:20:55.285 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-1882-gae248125-1jammy_all.deb ... 2024-05-15T12:20:55.433 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:20:56.537 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cachetools. 2024-05-15T12:20:56.565 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-05-15T12:20:56.604 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-05-15T12:20:56.865 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pyasn1. 2024-05-15T12:20:56.893 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-05-15T12:20:56.931 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-05-15T12:20:57.276 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-05-15T12:20:57.304 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-05-15T12:20:57.351 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-05-15T12:20:57.621 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rsa. 2024-05-15T12:20:57.648 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-05-15T12:20:57.687 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rsa (4.8-1) ... 2024-05-15T12:20:57.958 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-google-auth. 2024-05-15T12:20:57.986 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-05-15T12:20:58.024 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-05-15T12:20:58.336 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-05-15T12:20:58.363 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-05-15T12:20:58.402 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-15T12:20:58.722 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-websocket. 2024-05-15T12:20:58.750 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-05-15T12:20:58.788 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-05-15T12:20:59.100 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-kubernetes. 2024-05-15T12:20:59.128 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-05-15T12:20:59.174 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-15T12:20:59.830 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-05-15T12:20:59.859 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-1882-gae248125-1jammy_all.deb ... 2024-05-15T12:20:59.896 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:00.333 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libonig5:amd64. 2024-05-15T12:21:00.361 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-05-15T12:21:00.399 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-15T12:21:00.694 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libjq1:amd64. 2024-05-15T12:21:00.722 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-05-15T12:21:00.760 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-15T12:21:00.988 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package jq. 2024-05-15T12:21:01.005 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-05-15T12:21:01.046 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-05-15T12:21:01.299 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package socat. 2024-05-15T12:21:01.327 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-05-15T12:21:01.365 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-05-15T12:21:01.694 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package xmlstarlet. 2024-05-15T12:21:01.721 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-05-15T12:21:01.760 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-05-15T12:21:02.140 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-test. 2024-05-15T12:21:02.167 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../097-ceph-test_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:21:02.197 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-test (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:04.502 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-volume. 2024-05-15T12:21:04.530 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-1882-gae248125-1jammy_all.deb ... 2024-05-15T12:21:04.568 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-volume (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:04.855 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libcephfs-dev. 2024-05-15T12:21:04.873 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:21:04.913 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libcephfs-dev (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:05.233 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package lua-socket:amd64. 2024-05-15T12:21:05.261 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-05-15T12:21:05.407 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-15T12:21:05.754 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package lua-sec:amd64. 2024-05-15T12:21:05.782 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-05-15T12:21:05.820 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-05-15T12:21:06.116 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package nvme-cli. 2024-05-15T12:21:06.144 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-05-15T12:21:06.182 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-05-15T12:21:06.596 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package pkg-config. 2024-05-15T12:21:06.620 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-05-15T12:21:06.662 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-05-15T12:21:06.948 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-05-15T12:21:06.977 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-05-15T12:21:07.015 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-05-15T12:21:07.443 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-iniconfig. 2024-05-15T12:21:07.472 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-05-15T12:21:07.509 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-05-15T12:21:07.812 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pastescript. 2024-05-15T12:21:07.841 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-05-15T12:21:07.879 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-05-15T12:21:08.207 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pluggy. 2024-05-15T12:21:08.224 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-05-15T12:21:08.265 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-05-15T12:21:08.576 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-psutil. 2024-05-15T12:21:08.605 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-05-15T12:21:08.643 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-05-15T12:21:08.979 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-py. 2024-05-15T12:21:09.008 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-05-15T12:21:09.054 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-py (1.10.0-1) ... 2024-05-15T12:21:09.416 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pygments. 2024-05-15T12:21:09.444 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-05-15T12:21:09.482 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-05-15T12:21:09.928 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pyinotify. 2024-05-15T12:21:09.956 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-05-15T12:21:09.994 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-05-15T12:21:10.314 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-toml. 2024-05-15T12:21:10.342 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-05-15T12:21:10.380 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-05-15T12:21:10.691 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pytest. 2024-05-15T12:21:10.719 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-05-15T12:21:10.757 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-05-15T12:21:11.086 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-simplejson. 2024-05-15T12:21:11.115 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-05-15T12:21:11.152 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-05-15T12:21:11.422 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-05-15T12:21:11.438 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-05-15T12:21:11.480 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-05-15T12:21:12.417 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package radosgw. 2024-05-15T12:21:12.446 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../116-radosgw_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:21:12.483 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking radosgw (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:13.295 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package rbd-fuse. 2024-05-15T12:21:13.312 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-1882-gae248125-1jammy_amd64.deb ... 2024-05-15T12:21:13.353 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking rbd-fuse (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:13.658 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-05-15T12:21:14.008 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-15T12:21:14.108 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-05-15T12:21:14.346 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-05-15T12:21:14.657 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-05-15T12:21:15.061 INFO:teuthology.orchestra.run.smithi005.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-05-15T12:21:15.128 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up cephadm (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:15.552 INFO:teuthology.orchestra.run.smithi005.stdout:Adding system user cephadm....done 2024-05-15T12:21:15.667 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-15T12:21:15.943 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-05-15T12:21:16.211 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-05-15T12:21:16.337 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-05-15T12:21:16.596 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-05-15T12:21:16.873 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-05-15T12:21:16.974 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-py (1.10.0-1) ... 2024-05-15T12:21:17.275 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-05-15T12:21:17.628 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-05-15T12:21:17.887 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-05-15T12:21:18.163 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-05-15T12:21:18.414 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-05-15T12:21:18.657 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-05-15T12:21:18.900 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-05-15T12:21:19.142 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-ceph-argparse (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:19.394 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-15T12:21:19.520 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-15T12:21:19.802 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-05-15T12:21:19.930 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-05-15T12:21:20.189 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-05-15T12:21:20.546 INFO:teuthology.orchestra.run.smithi005.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-05-15T12:21:20.578 INFO:teuthology.orchestra.run.smithi005.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-05-15T12:21:20.634 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-15T12:21:20.735 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-05-15T12:21:21.103 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-15T12:21:21.229 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-05-15T12:21:21.497 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-05-15T12:21:21.773 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-05-15T12:21:22.074 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up zip (3.0-12build2) ... 2024-05-15T12:21:22.208 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-05-15T12:21:22.894 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-05-15T12:21:23.153 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-05-15T12:21:23.279 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-05-15T12:21:23.405 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-15T12:21:23.706 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-05-15T12:21:24.090 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-05-15T12:21:24.333 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-05-15T12:21:24.693 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-15T12:21:25.002 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-15T12:21:25.362 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-05-15T12:21:25.622 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-05-15T12:21:25.768 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-ceph-common (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:26.074 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-idna (3.3-1) ... 2024-05-15T12:21:26.383 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-15T12:21:27.605 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-05-15T12:21:27.849 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-15T12:21:27.966 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-toml (0.10.2-1) ... 2024-05-15T12:21:28.217 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-15T12:21:28.327 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-05-15T12:21:28.453 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-15T12:21:28.754 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-05-15T12:21:29.013 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-05-15T12:21:29.273 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-15T12:21:29.374 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-05-15T12:21:29.666 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-05-15T12:21:29.918 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-05-15T12:21:30.194 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-05-15T12:21:30.454 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-05-15T12:21:30.704 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-05-15T12:21:30.964 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-15T12:21:31.073 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-05-15T12:21:31.332 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-15T12:21:31.433 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-15T12:21:31.726 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-05-15T12:21:32.028 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-05-15T12:21:32.287 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-15T12:21:32.430 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-05-15T12:21:32.556 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-05-15T12:21:32.645 INFO:teuthology.orchestra.run.smithi005.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-05-15T12:21:32.882 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-15T12:21:32.991 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-05-15T12:21:33.376 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-05-15T12:21:33.655 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up lua-any (27ubuntu1) ... 2024-05-15T12:21:33.797 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-portend (3.0.0-1) ... 2024-05-15T12:21:34.056 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-15T12:21:34.157 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-15T12:21:34.459 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-05-15T12:21:34.761 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-15T12:21:35.036 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-05-15T12:21:35.162 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rsa (4.8-1) ... 2024-05-15T12:21:35.463 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-05-15T12:21:35.739 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-15T12:21:36.092 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-05-15T12:21:36.469 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-05-15T12:21:36.754 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-05-15T12:21:37.107 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-15T12:21:37.216 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librados2 (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:37.317 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:37.426 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-05-15T12:21:37.804 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libcephfs2 (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:37.930 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libradosstriper1 (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:38.056 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-05-15T12:21:38.333 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librbd1 (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:38.459 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-modules-core (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:38.585 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-fuse (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:38.806 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-15T12:21:38.806 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-15T12:21:39.323 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libcephfs-dev (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:39.504 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rados (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:39.639 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librgw2 (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:39.756 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-15T12:21:41.019 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rbd (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:41.145 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up rbd-fuse (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:41.271 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rgw (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:41.414 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cephfs (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:41.540 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-common (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:42.049 INFO:teuthology.orchestra.run.smithi005.stdout:Adding group ceph....done 2024-05-15T12:21:42.352 INFO:teuthology.orchestra.run.smithi005.stdout:Adding system user ceph....done 2024-05-15T12:21:42.487 INFO:teuthology.orchestra.run.smithi005.stdout:Setting system user ceph properties....done 2024-05-15T12:21:42.501 INFO:teuthology.orchestra.run.smithi005.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-05-15T12:21:42.649 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-05-15T12:21:42.964 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-05-15T12:21:43.450 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-test (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:43.569 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up radosgw (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:44.008 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-15T12:21:44.008 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-15T12:21:44.469 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-base (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:44.830 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-05-15T12:21:45.334 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mds (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:45.673 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-15T12:21:45.674 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-15T12:21:46.140 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:46.335 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-15T12:21:46.335 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-15T12:21:46.826 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-osd (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:47.118 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-15T12:21:47.118 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-15T12:21:47.585 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:47.711 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:48.259 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mon (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:48.459 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-15T12:21:48.460 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-15T12:21:48.942 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-cephadm (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:49.102 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:49.228 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-dashboard (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:49.387 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-volume (19.0.0-1882-gae248125-1jammy) ... 2024-05-15T12:21:49.782 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-05-15T12:21:49.899 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-05-15T12:21:50.035 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-05-15T12:21:50.802 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-05-15T12:21:51.465 INFO:teuthology.orchestra.run.smithi005.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-05-15T12:21:51.470 DEBUG:teuthology.parallel:result is None 2024-05-15T12:21:51.470 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:21:51.690 DEBUG:teuthology.orchestra.run.smithi005:> dpkg-query -W -f '${Version}' ceph 2024-05-15T12:21:51.710 INFO:teuthology.orchestra.run.smithi005.stdout:19.0.0-1882-gae248125-1jammy 2024-05-15T12:21:51.710 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1882-gae248125-1jammy 2024-05-15T12:21:51.710 INFO:teuthology.task.install:The correct ceph version 19.0.0-1882-gae248125-1jammy is installed. 2024-05-15T12:21:51.712 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-15T12:21:51.713 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-05-15T12:21:51.713 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-15T12:21:51.763 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-15T12:21:51.763 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-05-15T12:21:51.763 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/daemon-helper 2024-05-15T12:21:51.819 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-15T12:21:51.875 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-15T12:21:51.875 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-05-15T12:21:51.876 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-15T12:21:51.932 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-15T12:21:51.990 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-15T12:21:51.991 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-05-15T12:21:51.991 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/stdin-killer 2024-05-15T12:21:52.043 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-15T12:21:52.099 INFO:teuthology.run_tasks:Running task workunit... 2024-05-15T12:21:52.111 INFO:tasks.workunit:Pulling workunits from ref ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:21:52.112 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-15T12:21:52.112 INFO:tasks.workunit:timeout=3h 2024-05-15T12:21:52.112 INFO:tasks.workunit:cleanup=True 2024-05-15T12:21:52.112 DEBUG:teuthology.orchestra.run.smithi005:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-15T12:21:52.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:21:52.148 INFO:teuthology.orchestra.run.smithi005.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-15T12:21:52.148 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-05-15T12:21:52.199 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-05-15T12:21:52.199 DEBUG:teuthology.orchestra.run.smithi005:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-05-15T12:21:52.248 DEBUG:teuthology.orchestra.run.smithi005:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:21:52.298 INFO:tasks.workunit.client.0.smithi005.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-15T12:23:22.179 INFO:tasks.workunit.client.0.smithi005.stderr:Note: switching to 'ae2481253f9b2fa3451954e615eedb67ebdd5314'. 2024-05-15T12:23:22.179 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-05-15T12:23:22.180 INFO:tasks.workunit.client.0.smithi005.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-15T12:23:22.180 INFO:tasks.workunit.client.0.smithi005.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-15T12:23:22.180 INFO:tasks.workunit.client.0.smithi005.stderr:state without impacting any branches by switching back to a branch. 2024-05-15T12:23:22.180 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-05-15T12:23:22.180 INFO:tasks.workunit.client.0.smithi005.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-15T12:23:22.180 INFO:tasks.workunit.client.0.smithi005.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-15T12:23:22.180 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-05-15T12:23:22.180 INFO:tasks.workunit.client.0.smithi005.stderr: git switch -c 2024-05-15T12:23:22.180 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-05-15T12:23:22.180 INFO:tasks.workunit.client.0.smithi005.stderr:Or undo this operation with: 2024-05-15T12:23:22.180 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-05-15T12:23:22.180 INFO:tasks.workunit.client.0.smithi005.stderr: git switch - 2024-05-15T12:23:22.180 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-05-15T12:23:22.180 INFO:tasks.workunit.client.0.smithi005.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-15T12:23:22.181 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-05-15T12:23:22.181 INFO:tasks.workunit.client.0.smithi005.stderr:HEAD is now at ae2481253f9 DNM: teuto retrigger 2024-05-15T12:23:22.183 DEBUG:teuthology.orchestra.run.smithi005:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-15T12:23:22.232 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-05-15T12:23:22.232 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-15T12:23:22.278 INFO:tasks.workunit:Running workunits matching erasure-code on client.0... 2024-05-15T12:23:22.279 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code-plugins.sh... 2024-05-15T12:23:22.279 DEBUG:teuthology.orchestra.run.smithi005:workunit test erasure-code/test-erasure-code-plugins.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae2481253f9b2fa3451954e615eedb67ebdd5314 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh 2024-05-15T12:23:22.331 INFO:tasks.workunit.client.0.smithi005.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-05-15T12:23:22.331 INFO:tasks.workunit.client.0.smithi005.stderr:++ TIMEOUT=300 2024-05-15T12:23:22.331 INFO:tasks.workunit.client.0.smithi005.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-05-15T12:23:22.331 INFO:tasks.workunit.client.0.smithi005.stderr:++ MAX_TIMEOUT=15 2024-05-15T12:23:22.331 INFO:tasks.workunit.client.0.smithi005.stderr:++ PG_NUM=4 2024-05-15T12:23:22.332 INFO:tasks.workunit.client.0.smithi005.stderr:++ TMPDIR=/tmp 2024-05-15T12:23:22.332 INFO:tasks.workunit.client.0.smithi005.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-05-15T12:23:22.332 INFO:tasks.workunit.client.0.smithi005.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-05-15T12:23:22.332 INFO:tasks.workunit.client.0.smithi005.stderr:++ type xmlstarlet 2024-05-15T12:23:22.332 INFO:tasks.workunit.client.0.smithi005.stderr:++ XMLSTARLET=xmlstarlet 2024-05-15T12:23:22.332 INFO:tasks.workunit.client.0.smithi005.stderr:+++ uname 2024-05-15T12:23:22.333 INFO:tasks.workunit.client.0.smithi005.stderr:++ '[' Linux = FreeBSD ']' 2024-05-15T12:23:22.333 INFO:tasks.workunit.client.0.smithi005.stderr:++ SED=sed 2024-05-15T12:23:22.333 INFO:tasks.workunit.client.0.smithi005.stderr:++ AWK=awk 2024-05-15T12:23:22.333 INFO:tasks.workunit.client.0.smithi005.stderr:+++ stty -a 2024-05-15T12:23:22.333 INFO:tasks.workunit.client.0.smithi005.stderr:+++ head -1 2024-05-15T12:23:22.333 INFO:tasks.workunit.client.0.smithi005.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-05-15T12:23:22.345 INFO:tasks.workunit.client.0.smithi005.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-05-15T12:23:22.345 INFO:tasks.workunit.client.0.smithi005.stderr:++ termwidth= 2024-05-15T12:23:22.345 INFO:tasks.workunit.client.0.smithi005.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-05-15T12:23:22.345 INFO:tasks.workunit.client.0.smithi005.stderr:++ DIFFCOLOPTS='-y ' 2024-05-15T12:23:22.345 INFO:tasks.workunit.client.0.smithi005.stderr:++ KERNCORE=kernel.core_pattern 2024-05-15T12:23:22.346 INFO:tasks.workunit.client.0.smithi005.stderr:++ EXTRA_OPTS= 2024-05-15T12:23:22.349 INFO:tasks.workunit.client.0.smithi005.stderr:++ test '' = TESTS 2024-05-15T12:23:22.349 INFO:tasks.workunit.client.0.smithi005.stderr:++ uname -m 2024-05-15T12:23:22.351 INFO:tasks.workunit.client.0.smithi005.stderr:+ arch=x86_64 2024-05-15T12:23:22.351 INFO:tasks.workunit.client.0.smithi005.stderr:+ case $arch in 2024-05-15T12:23:22.351 INFO:tasks.workunit.client.0.smithi005.stderr:+ legacy_jerasure_plugins=(jerasure_generic jerasure_sse3 jerasure_sse4) 2024-05-15T12:23:22.351 INFO:tasks.workunit.client.0.smithi005.stderr:+ legacy_shec_plugins=(shec_generic shec_sse3 shec_sse4) 2024-05-15T12:23:22.351 INFO:tasks.workunit.client.0.smithi005.stderr:+ plugins=(jerasure shec lrc isa) 2024-05-15T12:23:22.351 INFO:tasks.workunit.client.0.smithi005.stderr:+ main test-erasure-code-plugins 2024-05-15T12:23:22.351 INFO:tasks.workunit.client.0.smithi005.stderr:+ local dir=td/test-erasure-code-plugins 2024-05-15T12:23:22.351 INFO:tasks.workunit.client.0.smithi005.stderr:+ shift 2024-05-15T12:23:22.351 INFO:tasks.workunit.client.0.smithi005.stderr:+ shopt -s -o xtrace 2024-05-15T12:23:22.351 INFO:tasks.workunit.client.0.smithi005.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-05-15T12:23:22.351 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-05-15T12:23:22.351 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-05-15T12:23:22.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-05-15T12:23:22.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-05-15T12:23:22.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-05-15T12:23:22.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-05-15T12:23:22.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-05-15T12:23:22.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-05-15T12:23:22.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/test-erasure-code-plugins 2024-05-15T12:23:22.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:26: run: local dir=td/test-erasure-code-plugins 2024-05-15T12:23:22.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:27: run: shift 2024-05-15T12:23:22.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: export CEPH_MON=127.0.0.1:17110 2024-05-15T12:23:22.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: CEPH_MON=127.0.0.1:17110 2024-05-15T12:23:22.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:30: run: export CEPH_ARGS 2024-05-15T12:23:22.353 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: uuidgen 2024-05-15T12:23:22.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: CEPH_ARGS+='--fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none ' 2024-05-15T12:23:22.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:17110 ' 2024-05-15T12:23:22.354 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: set 2024-05-15T12:23:22.354 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-05-15T12:23:22.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: local 'funcs=TEST_ec_profile_warning 2024-05-15T12:23:22.356 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_preload_no_warning 2024-05-15T12:23:22.356 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_preload_no_warning_default 2024-05-15T12:23:22.356 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_preload_warning' 2024-05-15T12:23:22.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-05-15T12:23:22.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_ec_profile_warning td/test-erasure-code-plugins 2024-05-15T12:23:22.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:92: TEST_ec_profile_warning: local dir=td/test-erasure-code-plugins 2024-05-15T12:23:22.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:94: TEST_ec_profile_warning: setup td/test-erasure-code-plugins 2024-05-15T12:23:22.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-15T12:23:22.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-15T12:23:22.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:23:22.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:23:22.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:23:22.356 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:23:22.356 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:23:22.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:23:22.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:23:22.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:23:22.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:23:22.358 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:23:22.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:23:22.359 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:23:22.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:23:22.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:23:22.360 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:23:22.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:23:22.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:23:22.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:23:22.361 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:23:22.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:23:22.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:23:22.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:23:22.364 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:23:22.365 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:23:22.365 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:23:22.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:23:22.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:23:22.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:23:22.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-15T12:23:22.367 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T12:23:22.367 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:23:22.367 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:23:22.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20204 2024-05-15T12:23:22.369 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T12:23:22.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-05-15T12:23:22.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-05-15T12:23:22.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T12:23:22.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-15T12:23:22.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:95: TEST_ec_profile_warning: run_mon td/test-erasure-code-plugins a 2024-05-15T12:23:22.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-15T12:23:22.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T12:23:22.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T12:23:22.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T12:23:22.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-15T12:23:22.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2024-05-15T12:23:22.877 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T12:23:22.877 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:23:22.877 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:23:22.878 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:23:22.878 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:23:22.878 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:23:22.878 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:23:22.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-15T12:23:23.094 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T12:23:23.095 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T12:23:23.095 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:23:23.096 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:23:23.096 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T12:23:23.097 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T12:23:23.097 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:23:23.097 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:23:23.097 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:23:23.097 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:23:23.097 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:23:23.097 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:23:23.097 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:23:23.097 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:23:23.098 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get fsid 2024-05-15T12:23:23.212 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T12:23:23.212 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:23:23.212 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:23:23.212 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T12:23:23.213 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T12:23:23.213 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:23:23.213 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:23:23.213 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:23:23.214 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:23:23.214 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:23:23.214 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:23:23.214 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:23:23.215 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:23:23.215 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get mon_host 2024-05-15T12:23:23.326 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:96: TEST_ec_profile_warning: run_mgr td/test-erasure-code-plugins x 2024-05-15T12:23:23.326 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-15T12:23:23.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T12:23:23.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T12:23:23.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T12:23:23.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-15T12:23:23.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T12:23:24.272 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T12:23:24.272 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:23:24.272 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:23:24.272 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:23:24.272 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:23:24.272 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:23:24.272 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:23:24.272 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:23:24.284 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:23:24.410 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: seq 0 2 2024-05-15T12:23:24.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-05-15T12:23:24.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 0 2024-05-15T12:23:24.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-15T12:23:24.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:23:24.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T12:23:24.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:23:24.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-15T12:23:24.415 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-15T12:23:24.415 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:23:24.415 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:23:24.415 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:23:24.415 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-15T12:23:24.415 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-15T12:23:24.415 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:23:24.415 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:23:24.415 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-15T12:23:24.415 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:23:24.415 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:23:24.415 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:23:24.416 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:23:24.416 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:23:24.416 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:23:24.416 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:23:24.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:23:24.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:23:24.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:23:24.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:23:24.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-15T12:23:24.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-15T12:23:24.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:23:24.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:23:24.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:23:24.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:23:24.417 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:23:24.417 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:23:24.417 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-15T12:23:24.417 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:23:24.417 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1720ff89-e60e-4fe5-98ed-572b417c252b 2024-05-15T12:23:24.417 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 1720ff89-e60e-4fe5-98ed-572b417c252b 2024-05-15T12:23:24.418 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1720ff89-e60e-4fe5-98ed-572b417c252b' 2024-05-15T12:23:24.418 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:23:24.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA8qURmsQ9fGhAA1ZuNccKEkRsouuQ8FxfQrA== 2024-05-15T12:23:24.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA8qURmsQ9fGhAA1ZuNccKEkRsouuQ8FxfQrA=="}' 2024-05-15T12:23:24.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1720ff89-e60e-4fe5-98ed-572b417c252b -i td/test-erasure-code-plugins/0/new.json 2024-05-15T12:23:24.758 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:23:24.776 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-15T12:23:24.777 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA8qURmsQ9fGhAA1ZuNccKEkRsouuQ8FxfQrA== --osd-uuid 1720ff89-e60e-4fe5-98ed-572b417c252b 2024-05-15T12:23:24.902 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:23:24.911+0000 7f476cb3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:23:24.935 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:23:24.943+0000 7f476cb3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:23:24.937 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:23:24.943+0000 7f476cb3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:23:24.938 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:23:24.947+0000 7f476cb3e7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:23:24.938 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:23:24.947+0000 7f476cb3e7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:23:24.938 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:23:24.947+0000 7f476cb3e7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:23:25.102 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:23:25.107+0000 7f476cb3e7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-15T12:23:31.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-15T12:23:31.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:23:31.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T12:23:31.359 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T12:23:31.359 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:23:31.603 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T12:23:31.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T12:23:31.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:23:31.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:23:31.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:23:31.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:23:31.635 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:23:31.643+0000 7fd2f40117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:23:31.638 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:23:31.643+0000 7fd2f40117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:23:31.639 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:23:31.647+0000 7fd2f40117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:23:31.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T12:23:31.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:23:31.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T12:23:31.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:23:31.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:23:31.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:23:31.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:23:31.824 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:23:31.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:23:31.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:23:32.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:23:32.192 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:23:32.199+0000 7fd2f40117c0 -1 Falling back to public interface 2024-05-15T12:23:33.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:23:33.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:23:33.057 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:23:33.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:23:33.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:23:33.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:23:33.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:23:34.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:23:34.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:23:34.288 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:23:34.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:23:34.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:23:34.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:23:34.698 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:23:34.869 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:23:34.875+0000 7fd2f40117c0 -1 osd.0 0 log_to_monitors true 2024-05-15T12:23:35.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:23:35.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:23:35.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:23:35.701 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:23:35.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:23:35.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:23:36.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:23:37.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:23:37.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:23:37.128 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:23:37.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:23:37.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:23:37.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:23:37.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:23:38.559 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:23:38.559 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:23:38.560 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:23:38.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:23:38.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:23:38.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:23:39.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:23:40.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:23:40.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:23:40.135 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:23:40.136 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:23:40.136 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:23:40.136 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:23:40.588 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:23:41.589 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:23:41.589 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:23:41.589 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:23:41.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:23:41.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:23:41.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:23:42.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:23:43.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:23:43.174 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:23:43.174 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:23:43.174 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:23:43.174 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:23:43.174 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:23:43.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:23:44.642 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:23:44.642 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:23:44.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:23:44.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:23:44.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:23:44.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:23:45.143 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:23:46.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:23:46.145 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:23:46.145 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:23:46.145 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:23:46.145 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:23:46.145 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:23:46.621 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:23:47.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:23:47.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:23:47.623 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:23:47.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:23:47.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:23:47.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:23:48.154 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:23:49.156 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:23:49.156 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:23:49.156 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:23:49.157 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:23:49.157 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:23:49.157 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:23:49.687 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:23:50.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:23:50.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:23:50.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:23:50.689 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:23:50.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:23:50.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:23:51.785 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:23:52.786 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:23:52.786 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:23:52.787 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:23:52.787 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:23:52.787 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:23:52.787 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:23:53.284 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:23:54.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:23:54.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:23:54.285 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:23:54.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:23:54.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:23:54.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:23:55.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:23:56.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:23:56.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:23:56.823 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:23:56.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:23:56.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:23:56.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:23:58.036 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:23:59.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:23:59.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:23:59.038 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:23:59.038 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:23:59.038 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:23:59.038 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:23:59.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:00.608 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:00.609 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:00.609 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T12:24:00.609 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T12:24:00.610 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:00.610 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:24:01.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:02.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:02.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:02.292 INFO:tasks.workunit.client.0.smithi005.stdout:19 2024-05-15T12:24:02.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-15T12:24:02.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:02.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:24:02.722 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3253420136,v1:127.0.0.1:6801/3253420136] [v2:127.0.0.1:6802/3253420136,v1:127.0.0.1:6803/3253420136] exists,up 1720ff89-e60e-4fe5-98ed-572b417c252b 2024-05-15T12:24:02.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:24:02.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:24:02.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:24:02.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-05-15T12:24:02.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 1 2024-05-15T12:24:02.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-15T12:24:02.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:24:02.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-15T12:24:02.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:24:02.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/1 2024-05-15T12:24:02.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-15T12:24:02.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:24:02.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:24:02.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:24:02.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/1' 2024-05-15T12:24:02.732 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/1/journal' 2024-05-15T12:24:02.732 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:24:02.732 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:24:02.732 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-15T12:24:02.732 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:24:02.732 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:24:02.733 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:24:02.733 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:24:02.733 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:24:02.733 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:24:02.733 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:24:02.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:24:02.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:24:02.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:24:02.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:24:02.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-15T12:24:02.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-15T12:24:02.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:24:02.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:24:02.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:24:02.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:24:02.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:24:02.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:24:02.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/1 2024-05-15T12:24:02.735 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:24:02.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b4be7052-c154-45dd-ae24-633dd549416f 2024-05-15T12:24:02.735 INFO:tasks.workunit.client.0.smithi005.stdout:add osd1 b4be7052-c154-45dd-ae24-633dd549416f 2024-05-15T12:24:02.736 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b4be7052-c154-45dd-ae24-633dd549416f' 2024-05-15T12:24:02.736 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:24:02.744 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBiqURmliPnLBAA+JyyKmNmtY67sBYUBcYVvw== 2024-05-15T12:24:02.744 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBiqURmliPnLBAA+JyyKmNmtY67sBYUBcYVvw=="}' 2024-05-15T12:24:02.744 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b4be7052-c154-45dd-ae24-633dd549416f -i td/test-erasure-code-plugins/1/new.json 2024-05-15T12:24:03.343 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:24:03.359 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/1/new.json 2024-05-15T12:24:03.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBiqURmliPnLBAA+JyyKmNmtY67sBYUBcYVvw== --osd-uuid b4be7052-c154-45dd-ae24-633dd549416f 2024-05-15T12:24:03.387 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:03.395+0000 7fc722c787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:24:03.421 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:03.427+0000 7fc722c787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:24:03.422 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:03.431+0000 7fc722c787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:24:03.422 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:03.431+0000 7fc722c787c0 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-05-15T12:24:03.422 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:03.431+0000 7fc722c787c0 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-05-15T12:24:03.422 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:03.431+0000 7fc722c787c0 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-05-15T12:24:03.501 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:03.507+0000 7fc722c787c0 -1 bluestore(td/test-erasure-code-plugins/1) _read_fsid unparsable uuid 2024-05-15T12:24:09.910 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/1/keyring 2024-05-15T12:24:09.911 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:24:09.911 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd1 key to auth repository 2024-05-15T12:24:09.911 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-15T12:24:09.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:24:10.485 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.1 2024-05-15T12:24:10.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-15T12:24:10.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:24:10.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:24:10.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:24:10.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:24:10.515 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:10.523+0000 7f32a9a777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:24:10.650 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:10.659+0000 7f32a9a777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:24:10.652 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:10.659+0000 7f32a9a777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:24:10.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-15T12:24:10.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:24:10.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-15T12:24:10.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:24:10.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:24:10.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:10.957 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:24:10.958 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:24:10.958 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:10.958 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:24:11.203 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:11.211+0000 7f32a9a777c0 -1 Falling back to public interface 2024-05-15T12:24:11.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:12.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:12.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:12.391 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:24:12.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:24:12.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:12.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:24:12.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:13.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:13.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:13.815 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:24:13.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:24:13.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:13.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:24:13.907 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:13.915+0000 7f32a9a777c0 -1 osd.1 0 log_to_monitors true 2024-05-15T12:24:14.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:15.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:15.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:15.235 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:24:15.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:24:15.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:15.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:24:15.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:16.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:16.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:16.769 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:24:16.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:24:16.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:16.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:24:17.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:18.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:18.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:18.283 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:24:18.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:24:18.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:18.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:24:18.893 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:19.896 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:19.896 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:19.896 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:24:19.896 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:24:19.897 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:19.897 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:24:20.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:21.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:21.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:21.367 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:24:21.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:24:21.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:21.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:24:21.817 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:22.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:22.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:22.819 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:24:22.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:24:22.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:22.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:24:23.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:24.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:24.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:24.244 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:24:24.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:24:24.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:24.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:24:24.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:25.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:25.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:25.765 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:24:25.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:24:25.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:25.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:24:26.220 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:27.222 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:27.222 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:27.223 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:24:27.223 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:24:27.223 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:27.223 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:24:27.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:28.861 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:28.862 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:28.862 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:24:28.862 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:24:28.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:28.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:24:29.442 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:30.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:30.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:30.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:24:30.445 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:24:30.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:30.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:24:31.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:32.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:32.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:32.175 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:24:32.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:24:32.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:32.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:24:33.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:34.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:34.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:34.004 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:24:34.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:24:34.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:34.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:24:34.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:35.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:35.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:35.877 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:24:35.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:24:35.878 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:35.878 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:24:36.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:37.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:37.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:37.872 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:24:37.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:24:37.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:37.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:24:38.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:39.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:39.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:39.429 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T12:24:39.430 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T12:24:39.430 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:39.430 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:24:39.921 INFO:tasks.workunit.client.0.smithi005.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/503653515,v1:127.0.0.1:6811/503653515] [v2:127.0.0.1:6812/503653515,v1:127.0.0.1:6813/503653515] exists,up b4be7052-c154-45dd-ae24-633dd549416f 2024-05-15T12:24:39.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:24:39.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:24:39.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:24:39.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-05-15T12:24:39.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 2 2024-05-15T12:24:39.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-15T12:24:39.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:24:39.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-15T12:24:39.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:24:39.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/2 2024-05-15T12:24:39.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-15T12:24:39.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:24:39.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:24:39.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:24:39.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/2' 2024-05-15T12:24:39.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/2/journal' 2024-05-15T12:24:39.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:24:39.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:24:39.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-15T12:24:39.926 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:24:39.926 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:24:39.926 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:24:39.926 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:24:39.926 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:24:39.926 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:24:39.926 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:24:39.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:24:39.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:24:39.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:24:39.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:24:39.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-15T12:24:39.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-15T12:24:39.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:24:39.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:24:39.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:24:39.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:24:39.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:24:39.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:24:39.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/2 2024-05-15T12:24:39.928 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:24:39.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=07b94712-d06b-4e39-a9d7-e46f9e58afa1 2024-05-15T12:24:39.928 INFO:tasks.workunit.client.0.smithi005.stdout:add osd2 07b94712-d06b-4e39-a9d7-e46f9e58afa1 2024-05-15T12:24:39.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 07b94712-d06b-4e39-a9d7-e46f9e58afa1' 2024-05-15T12:24:39.928 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:24:39.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCHqURmeqDnOBAAL5kZwanZCFrG2BjcWQVf7w== 2024-05-15T12:24:39.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCHqURmeqDnOBAAL5kZwanZCFrG2BjcWQVf7w=="}' 2024-05-15T12:24:39.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 07b94712-d06b-4e39-a9d7-e46f9e58afa1 -i td/test-erasure-code-plugins/2/new.json 2024-05-15T12:24:40.550 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:24:40.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/2/new.json 2024-05-15T12:24:40.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHqURmeqDnOBAAL5kZwanZCFrG2BjcWQVf7w== --osd-uuid 07b94712-d06b-4e39-a9d7-e46f9e58afa1 2024-05-15T12:24:40.604 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:40.611+0000 7f513bc827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:24:40.644 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:40.651+0000 7f513bc827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:24:40.646 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:40.655+0000 7f513bc827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:24:40.647 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:40.655+0000 7f513bc827c0 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-05-15T12:24:40.647 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:40.655+0000 7f513bc827c0 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-05-15T12:24:40.647 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:40.655+0000 7f513bc827c0 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-05-15T12:24:40.675 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:40.683+0000 7f513bc827c0 -1 bluestore(td/test-erasure-code-plugins/2) _read_fsid unparsable uuid 2024-05-15T12:24:47.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/2/keyring 2024-05-15T12:24:47.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:24:47.332 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd2 key to auth repository 2024-05-15T12:24:47.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-15T12:24:47.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:24:47.891 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.2 2024-05-15T12:24:47.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-15T12:24:47.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:24:47.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:24:47.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:24:47.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:24:47.922 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:47.931+0000 7fd5a64847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:24:47.967 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:47.975+0000 7fd5a64847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:24:47.969 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:47.979+0000 7fd5a64847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:24:48.351 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-15T12:24:48.351 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:24:48.351 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-15T12:24:48.351 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:24:48.351 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:24:48.351 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:48.351 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:24:48.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:24:48.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:48.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:24:48.538 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:48.547+0000 7fd5a64847c0 -1 Falling back to public interface 2024-05-15T12:24:48.776 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:49.777 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:49.777 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:49.778 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:24:49.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:24:49.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:49.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:24:50.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:51.225 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:24:51.235+0000 7fd5a64847c0 -1 osd.2 0 log_to_monitors true 2024-05-15T12:24:51.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:51.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:51.259 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:24:51.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:24:51.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:24:51.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:51.726 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:52.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:52.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:52.728 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:24:52.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:24:52.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:52.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:24:53.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:54.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:54.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:54.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:24:54.282 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:24:54.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:54.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:24:54.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:55.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:55.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:55.730 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:24:55.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:24:55.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:55.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:24:56.210 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:57.212 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:57.212 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:57.212 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:24:57.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:24:57.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:57.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:24:57.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:24:58.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:24:58.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:24:58.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:24:58.768 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:24:58.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:24:58.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:24:59.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:00.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:00.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:00.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:25:00.488 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:25:00.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:00.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:25:00.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:01.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:01.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:01.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:25:01.965 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:25:01.965 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:01.965 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:25:02.512 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:03.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:03.514 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:03.514 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:25:03.514 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:25:03.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:03.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:25:04.226 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:05.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:05.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:05.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:25:05.229 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:25:05.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:05.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:25:06.226 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:07.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:07.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:07.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:25:07.228 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:25:07.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:07.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:25:07.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:08.747 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:08.747 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:08.747 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:25:08.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:25:08.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:08.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:25:09.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:10.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:10.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:10.496 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:25:10.497 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:25:10.497 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:10.497 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:25:11.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:12.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:12.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:12.594 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:25:12.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:25:12.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:12.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:25:13.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:14.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:14.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:14.491 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:25:14.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:25:14.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:14.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:25:15.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:16.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:16.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:16.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:25:16.481 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:25:16.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:16.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:25:16.894 INFO:tasks.workunit.client.0.smithi005.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1791325825,v1:127.0.0.1:6819/1791325825] [v2:127.0.0.1:6820/1791325825,v1:127.0.0.1:6821/1791325825] exists,up 07b94712-d06b-4e39-a9d7-e46f9e58afa1 2024-05-15T12:25:16.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:25:16.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:25:16.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:25:16.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:100: TEST_ec_profile_warning: create_rbd_pool 2024-05-15T12:25:16.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-15T12:25:17.392 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd' does not exist 2024-05-15T12:25:17.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-15T12:25:17.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-15T12:25:17.973 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd' already exists 2024-05-15T12:25:17.991 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T12:25:18.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-15T12:25:20.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:101: TEST_ec_profile_warning: wait_for_clean 2024-05-15T12:25:20.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T12:25:20.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T12:25:20.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T12:25:20.729 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T12:25:20.729 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T12:25:20.730 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T12:25:20.730 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T12:25:20.730 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T12:25:20.730 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T12:25:20.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T12:25:20.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T12:25:20.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T12:25:20.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T12:25:20.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T12:25:20.856 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T12:25:21.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T12:25:21.287 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:25:21.287 INFO:tasks.workunit.client.0.smithi005.stderr:2' 2024-05-15T12:25:21.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T12:25:21.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:25:21.288 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T12:25:21.459 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-05-15T12:25:21.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-05-15T12:25:21.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-05-15T12:25:21.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:25:21.535 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T12:25:21.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-05-15T12:25:21.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-05-15T12:25:21.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672972' 2024-05-15T12:25:21.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:25:21.628 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T12:25:21.780 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-05-15T12:25:21.780 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-05-15T12:25:21.780 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672972 2-64424509446' 2024-05-15T12:25:21.780 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:25:21.781 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-05-15T12:25:21.781 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:25:21.782 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T12:25:21.783 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-05-15T12:25:21.783 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:25:21.783 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 21474836503 2024-05-15T12:25:21.784 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-05-15T12:25:21.784 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-05-15T12:25:21.784 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T12:25:22.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836503 2024-05-15T12:25:22.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-15T12:25:23.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-15T12:25:23.189 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T12:25:23.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836503 2024-05-15T12:25:23.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:25:23.626 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-05-15T12:25:23.627 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:25:23.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T12:25:23.629 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-05-15T12:25:23.630 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:25:23.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-05-15T12:25:23.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-05-15T12:25:23.631 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 42949672972 2024-05-15T12:25:23.631 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T12:25:24.038 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-05-15T12:25:24.038 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:25:24.039 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-05-15T12:25:24.040 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:25:24.041 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T12:25:24.042 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-05-15T12:25:24.042 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:25:24.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-05-15T12:25:24.043 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 64424509446 2024-05-15T12:25:24.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-05-15T12:25:24.044 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T12:25:24.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509446 2024-05-15T12:25:24.491 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T12:25:24.492 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:25:24.492 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:25:25.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-15T12:25:25.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T12:25:25.234 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T12:25:25.235 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T12:25:25.235 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T12:25:25.235 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T12:25:25.235 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T12:25:25.235 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T12:25:25.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-15T12:25:25.707 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T12:25:25.708 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:25:25.708 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:25:26.240 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-15T12:25:26.240 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T12:25:26.240 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T12:25:26.240 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-05-15T12:25:26.240 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_generic crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_generic 2024-05-15T12:25:26.918 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-05-15T12:25:26.918 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:25:26.919 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:25:26.919 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:25:26.919 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:25:26.919 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:25:26.919 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:25:26.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-05-15T12:25:26.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-mon.a.asok log flush 2024-05-15T12:25:27.025 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:25:27.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2024-05-15T12:25:27.045 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:25:26.582+0000 7f2763522640 0 mon.a@0(leader).osd e23 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-05-15T12:25:27.046 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:25:26.906+0000 7f2763522640 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-05-15T12:25:27.046 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:25:26.906+0000 7f2763522640 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-05-15T12:25:27.046 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-05-15T12:25:27.046 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse3 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse3 2024-05-15T12:25:27.598 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-05-15T12:25:27.598 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:25:27.599 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:25:27.599 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:25:27.599 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:25:27.599 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:25:27.599 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:25:27.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-05-15T12:25:27.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-mon.a.asok log flush 2024-05-15T12:25:27.700 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:25:27.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2024-05-15T12:25:27.719 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:25:27.358+0000 7f2763522640 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-05-15T12:25:27.719 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:25:27.586+0000 7f2763522640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-05-15T12:25:27.719 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:25:27.586+0000 7f2763522640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-05-15T12:25:27.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-05-15T12:25:27.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse4 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse4 2024-05-15T12:25:28.393 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-05-15T12:25:28.393 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:25:28.394 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:25:28.394 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:25:28.394 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:25:28.394 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:25:28.394 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:25:28.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-05-15T12:25:28.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-mon.a.asok log flush 2024-05-15T12:25:28.492 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:25:28.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2024-05-15T12:25:28.514 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:25:28.150+0000 7f2763522640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-05-15T12:25:28.514 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:25:28.382+0000 7f2763522640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-05-15T12:25:28.514 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:25:28.382+0000 7f2763522640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-05-15T12:25:28.514 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-05-15T12:25:28.514 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_generic crush-failure-domain=osd plugin=shec_generic 2024-05-15T12:25:29.107 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-05-15T12:25:29.107 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:25:29.107 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:25:29.107 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:25:29.107 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:25:29.108 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:25:29.108 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:25:29.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-05-15T12:25:29.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-mon.a.asok log flush 2024-05-15T12:25:29.222 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:25:29.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2024-05-15T12:25:29.243 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:25:28.874+0000 7f2763522640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-05-15T12:25:29.243 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:25:29.094+0000 7f2763522640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-05-15T12:25:29.243 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:25:29.094+0000 7f2763522640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-05-15T12:25:29.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-05-15T12:25:29.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse3 crush-failure-domain=osd plugin=shec_sse3 2024-05-15T12:25:29.920 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-05-15T12:25:29.920 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:25:29.920 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:25:29.920 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:25:29.920 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:25:29.920 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:25:29.921 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:25:29.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-05-15T12:25:29.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-mon.a.asok log flush 2024-05-15T12:25:30.023 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:25:30.036 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2024-05-15T12:25:30.043 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:25:29.602+0000 7f2763522640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-05-15T12:25:30.044 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:25:29.910+0000 7f2763522640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-05-15T12:25:30.044 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:25:29.910+0000 7f2763522640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-05-15T12:25:30.044 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-05-15T12:25:30.044 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse4 crush-failure-domain=osd plugin=shec_sse4 2024-05-15T12:25:30.653 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-05-15T12:25:30.653 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:25:30.653 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:25:30.654 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:25:30.654 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:25:30.654 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:25:30.654 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:25:30.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-05-15T12:25:30.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-mon.a.asok log flush 2024-05-15T12:25:30.760 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:25:30.773 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2024-05-15T12:25:30.782 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:25:30.370+0000 7f2763522640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-05-15T12:25:30.782 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:25:30.642+0000 7f2763522640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-05-15T12:25:30.782 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:25:30.642+0000 7f2763522640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-05-15T12:25:30.783 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:115: TEST_ec_profile_warning: teardown td/test-erasure-code-plugins 2024-05-15T12:25:30.783 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:25:30.783 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:25:30.783 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:25:30.784 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:25:30.784 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:25:30.785 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:25:30.785 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:25:30.785 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:25:30.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:25:30.895 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:25:30.896 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:25:30.896 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:25:30.897 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:25:30.897 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:25:30.898 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:25:30.898 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:25:30.898 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:25:30.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:25:30.899 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:25:30.900 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:25:30.900 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:25:30.900 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:25:30.919 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:25:30.919 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:25:30.919 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:25:30.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:25:30.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:25:30.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:25:30.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-05-15T12:25:30.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning td/test-erasure-code-plugins 2024-05-15T12:25:30.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:58: TEST_preload_no_warning: local dir=td/test-erasure-code-plugins 2024-05-15T12:25:30.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-05-15T12:25:30.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-05-15T12:25:30.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-15T12:25:30.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-15T12:25:30.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:25:30.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:25:30.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:25:30.921 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:25:30.922 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:25:30.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:25:30.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:25:30.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:25:30.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:25:30.923 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:25:30.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:25:30.924 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:25:30.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:25:30.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:25:30.925 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:25:30.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:25:30.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:25:30.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:25:30.927 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:25:30.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:25:30.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:25:30.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:25:30.929 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:25:30.929 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:25:30.929 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:25:30.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:25:30.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:25:30.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:25:30.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-15T12:25:30.931 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T12:25:30.932 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:25:30.932 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:25:30.932 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20204 2024-05-15T12:25:30.933 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T12:25:30.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-15T12:25:30.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T12:25:30.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-15T12:25:30.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure 2024-05-15T12:25:30.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-15T12:25:30.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T12:25:30.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T12:25:30.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T12:25:30.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-15T12:25:30.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure 2024-05-15T12:25:31.323 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T12:25:31.323 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:25:31.323 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:25:31.323 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:25:31.323 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:25:31.323 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:25:31.324 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:25:31.324 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure 2024-05-15T12:25:31.525 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T12:25:31.526 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T12:25:31.527 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:25:31.527 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:25:31.527 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T12:25:31.527 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T12:25:31.527 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:25:31.528 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:25:31.528 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:25:31.528 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:25:31.528 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:25:31.528 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:25:31.529 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:25:31.529 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:25:31.529 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get fsid 2024-05-15T12:25:31.643 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T12:25:31.643 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:25:31.644 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:25:31.644 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T12:25:31.645 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T12:25:31.645 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:25:31.645 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:25:31.645 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:25:31.645 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:25:31.645 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:25:31.645 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:25:31.645 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:25:31.645 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:25:31.646 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get mon_host 2024-05-15T12:25:31.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-05-15T12:25:31.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-15T12:25:31.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T12:25:31.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T12:25:31.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T12:25:31.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-15T12:25:31.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T12:25:32.650 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T12:25:32.650 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:25:32.650 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:25:32.650 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:25:32.650 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:25:32.650 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:25:32.651 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:25:32.651 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:25:32.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:25:32.683 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-05-15T12:25:32.683 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:25:32.684 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:25:32.684 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:25:32.684 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:25:32.684 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:25:32.684 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:25:32.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-05-15T12:25:32.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-mon.a.asok log flush 2024-05-15T12:25:32.784 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:25:32.799 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure 2024-05-15T12:25:32.799 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-15T12:25:32.799 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:25:32.799 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T12:25:32.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:25:32.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-15T12:25:32.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-15T12:25:32.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:25:32.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:25:32.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:25:32.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-15T12:25:32.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-15T12:25:32.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:25:32.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:25:32.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-15T12:25:32.800 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:25:32.800 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:25:32.800 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:25:32.802 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:25:32.802 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:25:32.802 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:25:32.803 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:25:32.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:25:32.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:25:32.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:25:32.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:25:32.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-15T12:25:32.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-15T12:25:32.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:25:32.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:25:32.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:25:32.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:25:32.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:25:32.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure 2024-05-15T12:25:32.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-15T12:25:32.804 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:25:32.804 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=431fb6f6-18be-462e-a9aa-7d9dd890f6e3 2024-05-15T12:25:32.804 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 431fb6f6-18be-462e-a9aa-7d9dd890f6e3 2024-05-15T12:25:32.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 431fb6f6-18be-462e-a9aa-7d9dd890f6e3' 2024-05-15T12:25:32.805 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:25:32.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC8qURmSEaBMRAA80LRNhSdUQCH4g6S0eQ6Pg== 2024-05-15T12:25:32.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC8qURmSEaBMRAA80LRNhSdUQCH4g6S0eQ6Pg=="}' 2024-05-15T12:25:32.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 431fb6f6-18be-462e-a9aa-7d9dd890f6e3 -i td/test-erasure-code-plugins/0/new.json 2024-05-15T12:25:33.113 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:25:33.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-15T12:25:33.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure --mkfs --key AQC8qURmSEaBMRAA80LRNhSdUQCH4g6S0eQ6Pg== --osd-uuid 431fb6f6-18be-462e-a9aa-7d9dd890f6e3 2024-05-15T12:25:33.169 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:25:33.178+0000 7f7856ad87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:25:33.173 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:25:33.182+0000 7f7856ad87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:25:33.174 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:25:33.182+0000 7f7856ad87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:25:33.175 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:25:33.182+0000 7f7856ad87c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:25:33.175 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:25:33.182+0000 7f7856ad87c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:25:33.175 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:25:33.182+0000 7f7856ad87c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:25:33.213 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:25:33.222+0000 7f7856ad87c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-15T12:25:39.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-15T12:25:39.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:25:39.427 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T12:25:39.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T12:25:39.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:25:39.987 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T12:25:39.988 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T12:25:39.988 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure 2024-05-15T12:25:39.988 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:25:39.988 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:25:39.988 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:25:40.011 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:25:40.018+0000 7f923eb707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:25:40.030 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:25:40.038+0000 7f923eb707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:25:40.031 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:25:40.038+0000 7f923eb707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:25:40.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T12:25:40.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:25:40.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T12:25:40.445 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:25:40.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:25:40.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:25:40.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:40.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:25:40.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:40.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:25:40.589 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:25:40.598+0000 7f923eb707c0 -1 Falling back to public interface 2024-05-15T12:25:40.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:41.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:41.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:41.920 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:25:41.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:25:41.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:41.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:25:42.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:43.195 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:25:43.202+0000 7f923eb707c0 -1 osd.0 0 log_to_monitors true 2024-05-15T12:25:43.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:43.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:43.339 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:25:43.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:25:43.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:43.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:25:43.817 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:44.818 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:44.818 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:44.819 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:25:44.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:25:44.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:44.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:25:45.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:46.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:46.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:46.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:25:46.358 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:25:46.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:46.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:25:46.781 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:47.782 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:47.782 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:47.783 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:25:47.783 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:25:47.783 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:47.783 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:25:48.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:49.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:49.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:49.283 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:25:49.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:25:49.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:49.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:25:49.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:50.751 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:50.752 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:50.752 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:25:50.752 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:25:50.752 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:50.752 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:25:51.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:52.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:52.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:52.331 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:25:52.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:25:52.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:52.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:25:52.773 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:53.774 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:53.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:53.775 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:25:53.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:25:53.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:53.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:25:54.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:55.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:55.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:55.287 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:25:55.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:25:55.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:55.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:25:55.817 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:56.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:56.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:56.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:25:56.819 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:25:56.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:56.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:25:57.389 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:25:58.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:25:58.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:25:58.391 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:25:58.392 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:25:58.392 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:25:58.392 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:25:59.231 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:00.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:00.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:00.233 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:26:00.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:26:00.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:00.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:00.977 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:01.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:01.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:01.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:26:01.979 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:26:01.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:01.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:03.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:04.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:04.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:04.270 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:26:04.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:26:04.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:04.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:05.051 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:06.053 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:06.053 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:06.053 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:26:06.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:26:06.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:06.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:06.687 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:07.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:07.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:07.690 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:26:07.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:26:07.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:07.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:08.163 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1989189545,v1:127.0.0.1:6803/1989189545] [v2:127.0.0.1:6804/1989189545,v1:127.0.0.1:6805/1989189545] exists,up 431fb6f6-18be-462e-a9aa-7d9dd890f6e3 2024-05-15T12:26:08.164 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:26:08.164 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:26:08.164 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:26:08.165 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-05-15T12:26:08.165 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T12:26:08.165 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T12:26:08.165 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:26:08.165 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:26:08.165 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:26:08.165 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-osd.0.asok 2024-05-15T12:26:08.166 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-05-15T12:26:08.166 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-osd.0.asok log flush 2024-05-15T12:26:08.281 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:26:08.294 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-05-15T12:26:08.300 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-05-15T12:26:08.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-05-15T12:26:08.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:26:08.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:26:08.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:26:08.306 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:26:08.307 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:26:08.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:26:08.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:26:08.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:26:08.615 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:26:08.615 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:26:08.617 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:26:08.617 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:26:08.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:26:08.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:26:08.619 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:26:08.620 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:26:08.620 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:26:08.621 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:26:08.621 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:26:08.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:26:08.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:26:08.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:26:08.631 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:26:08.631 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:26:08.631 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:26:08.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:26:08.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:26:08.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:26:08.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-05-15T12:26:08.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-05-15T12:26:08.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-15T12:26:08.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-15T12:26:08.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:26:08.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:26:08.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:26:08.634 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:26:08.634 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:26:08.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:26:08.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:26:08.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:26:08.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:26:08.637 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:26:08.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:26:08.638 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:26:08.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:26:08.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:26:08.640 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:26:08.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:26:08.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:26:08.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:26:08.642 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:26:08.642 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:26:08.644 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:26:08.644 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:26:08.645 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:26:08.645 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:26:08.645 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:26:08.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:26:08.646 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:26:08.646 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:26:08.646 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-15T12:26:08.648 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T12:26:08.648 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:26:08.648 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:26:08.648 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20204 2024-05-15T12:26:08.650 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T12:26:08.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-15T12:26:08.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T12:26:08.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-15T12:26:08.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec 2024-05-15T12:26:08.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-15T12:26:08.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T12:26:08.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T12:26:08.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T12:26:08.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-15T12:26:08.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec 2024-05-15T12:26:09.028 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T12:26:09.028 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:26:09.028 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:26:09.028 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:26:09.029 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:26:09.029 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:26:09.029 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:26:09.029 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec 2024-05-15T12:26:09.239 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T12:26:09.241 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T12:26:09.241 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:26:09.241 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:26:09.241 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T12:26:09.242 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T12:26:09.242 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:26:09.242 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:26:09.242 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:26:09.242 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:26:09.242 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:26:09.242 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:26:09.243 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:26:09.243 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:26:09.243 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get fsid 2024-05-15T12:26:09.361 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T12:26:09.361 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:26:09.361 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:26:09.361 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T12:26:09.362 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T12:26:09.362 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:26:09.362 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:26:09.362 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:26:09.363 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:26:09.363 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:26:09.363 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:26:09.364 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:26:09.364 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:26:09.364 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get mon_host 2024-05-15T12:26:09.474 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-05-15T12:26:09.475 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-15T12:26:09.475 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T12:26:09.475 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T12:26:09.475 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T12:26:09.475 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-15T12:26:09.475 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T12:26:10.388 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T12:26:10.388 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:26:10.389 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:26:10.389 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:26:10.389 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:26:10.389 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:26:10.389 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:26:10.389 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:26:10.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:26:10.425 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-05-15T12:26:10.426 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:26:10.426 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:26:10.426 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:26:10.426 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:26:10.426 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:26:10.426 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:26:10.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-05-15T12:26:10.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-mon.a.asok log flush 2024-05-15T12:26:10.525 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:26:10.539 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec 2024-05-15T12:26:10.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-15T12:26:10.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:26:10.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T12:26:10.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:26:10.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-15T12:26:10.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-15T12:26:10.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:26:10.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:26:10.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:26:10.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-15T12:26:10.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-15T12:26:10.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:26:10.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:26:10.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-15T12:26:10.541 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:26:10.541 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:26:10.541 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:26:10.541 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:26:10.541 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:26:10.541 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:26:10.541 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:26:10.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:26:10.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:26:10.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:26:10.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:26:10.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-15T12:26:10.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-15T12:26:10.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:26:10.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:26:10.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:26:10.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:26:10.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:26:10.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec 2024-05-15T12:26:10.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-15T12:26:10.543 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:26:10.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0b20110a-2636-4600-a132-6e7b57c565d8 2024-05-15T12:26:10.544 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 0b20110a-2636-4600-a132-6e7b57c565d8 2024-05-15T12:26:10.544 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0b20110a-2636-4600-a132-6e7b57c565d8' 2024-05-15T12:26:10.544 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:26:10.559 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDiqURmY0H4IRAA7luuBGD0FYYnBupyodjShQ== 2024-05-15T12:26:10.559 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDiqURmY0H4IRAA7luuBGD0FYYnBupyodjShQ=="}' 2024-05-15T12:26:10.559 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0b20110a-2636-4600-a132-6e7b57c565d8 -i td/test-erasure-code-plugins/0/new.json 2024-05-15T12:26:10.876 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:26:10.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-15T12:26:10.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec --mkfs --key AQDiqURmY0H4IRAA7luuBGD0FYYnBupyodjShQ== --osd-uuid 0b20110a-2636-4600-a132-6e7b57c565d8 2024-05-15T12:26:10.931 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:10.938+0000 7fb2b99ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:26:10.953 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:10.962+0000 7fb2b99ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:26:10.954 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:10.962+0000 7fb2b99ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:26:10.954 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:10.962+0000 7fb2b99ae7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:26:10.955 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:10.962+0000 7fb2b99ae7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:26:10.955 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:10.962+0000 7fb2b99ae7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:26:10.993 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:11.002+0000 7fb2b99ae7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-15T12:26:17.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-15T12:26:17.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:26:17.298 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T12:26:17.298 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T12:26:17.298 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:26:17.853 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T12:26:17.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T12:26:17.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec 2024-05-15T12:26:17.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:26:17.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:26:17.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:26:17.885 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:17.894+0000 7fd4e566b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:26:17.902 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:17.910+0000 7fd4e566b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:26:17.902 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:17.910+0000 7fd4e566b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:26:18.274 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T12:26:18.274 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:26:18.274 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T12:26:18.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:26:18.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:26:18.275 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:26:18.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:18.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:26:18.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:18.276 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:18.457 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:18.466+0000 7fd4e566b7c0 -1 Falling back to public interface 2024-05-15T12:26:18.687 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:19.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:19.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:19.689 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:26:19.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:26:19.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:19.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:20.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:21.005 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:21.014+0000 7fd4e566b7c0 -1 osd.0 0 log_to_monitors true 2024-05-15T12:26:21.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:21.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:21.116 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:26:21.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:26:21.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:21.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:21.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:22.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:22.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:22.660 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:26:22.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:26:22.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:22.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:23.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:24.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:24.117 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:26:24.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:24.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:26:24.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:24.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:24.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:25.653 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:25.653 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:25.653 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:26:25.653 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:26:25.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:25.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:26.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:27.146 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:27.146 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:26:27.147 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:27.147 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:26:27.147 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:27.147 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:27.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:28.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:28.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:28.593 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:26:28.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:26:28.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:28.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:29.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:30.131 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:30.131 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:30.131 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:26:30.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:26:30.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:30.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:30.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:31.552 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:31.552 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:31.553 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:26:31.553 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:26:31.553 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:31.553 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:32.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:33.156 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:33.156 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:33.156 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:26:33.157 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:26:33.157 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:33.157 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:33.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:34.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:34.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:34.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:26:34.645 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:26:34.646 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:34.646 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:35.808 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:36.810 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:36.810 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:36.810 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:26:36.811 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:26:36.811 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:36.811 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:37.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:38.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:38.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:38.636 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:26:38.637 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:26:38.637 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:38.637 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:39.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:40.479 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:40.479 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:40.479 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:26:40.479 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:26:40.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:40.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:41.276 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:42.278 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:42.278 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:42.278 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:26:42.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:26:42.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:42.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:42.997 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:43.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:43.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:43.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:26:44.000 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:26:44.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:44.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:44.716 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:45.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:45.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:45.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:26:45.719 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:26:45.719 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:45.719 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:46.225 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/523223316,v1:127.0.0.1:6803/523223316] [v2:127.0.0.1:6804/523223316,v1:127.0.0.1:6805/523223316] exists,up 0b20110a-2636-4600-a132-6e7b57c565d8 2024-05-15T12:26:46.226 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:26:46.226 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:26:46.226 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:26:46.227 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-05-15T12:26:46.227 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T12:26:46.227 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T12:26:46.227 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:26:46.227 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:26:46.227 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:26:46.228 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-osd.0.asok 2024-05-15T12:26:46.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-05-15T12:26:46.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-osd.0.asok log flush 2024-05-15T12:26:46.341 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:26:46.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-05-15T12:26:46.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-05-15T12:26:46.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-05-15T12:26:46.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:26:46.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:26:46.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:26:46.364 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:26:46.364 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:26:46.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:26:46.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:26:46.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:26:46.474 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:26:46.474 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:26:46.474 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:26:46.475 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:26:46.476 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:26:46.476 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:26:46.476 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:26:46.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:26:46.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:26:46.478 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:26:46.478 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:26:46.479 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:26:46.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:26:46.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:26:46.486 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:26:46.486 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:26:46.486 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:26:46.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:26:46.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:26:46.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:26:46.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-05-15T12:26:46.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-05-15T12:26:46.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-15T12:26:46.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-15T12:26:46.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:26:46.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:26:46.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:26:46.489 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:26:46.489 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:26:46.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:26:46.490 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:26:46.490 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:26:46.490 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:26:46.490 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:26:46.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:26:46.491 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:26:46.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:26:46.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:26:46.493 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:26:46.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:26:46.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:26:46.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:26:46.495 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:26:46.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:26:46.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:26:46.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:26:46.497 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:26:46.497 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:26:46.497 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:26:46.497 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:26:46.498 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:26:46.498 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:26:46.498 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-15T12:26:46.500 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T12:26:46.500 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:26:46.500 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:26:46.500 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20204 2024-05-15T12:26:46.501 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T12:26:46.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-15T12:26:46.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T12:26:46.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-15T12:26:46.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=lrc 2024-05-15T12:26:46.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-15T12:26:46.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T12:26:46.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T12:26:46.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T12:26:46.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-15T12:26:46.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=lrc 2024-05-15T12:26:46.890 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T12:26:46.890 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:26:46.890 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:26:46.891 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:26:46.891 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:26:46.891 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:26:46.891 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:26:46.891 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=lrc 2024-05-15T12:26:47.092 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T12:26:47.094 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T12:26:47.094 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:26:47.094 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:26:47.094 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T12:26:47.095 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T12:26:47.095 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:26:47.095 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:26:47.095 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:26:47.095 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:26:47.095 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:26:47.095 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:26:47.096 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:26:47.096 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:26:47.096 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get fsid 2024-05-15T12:26:47.214 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T12:26:47.215 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:26:47.215 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:26:47.215 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T12:26:47.215 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T12:26:47.216 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:26:47.216 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:26:47.216 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:26:47.216 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:26:47.216 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:26:47.216 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:26:47.216 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:26:47.216 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:26:47.217 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get mon_host 2024-05-15T12:26:47.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-05-15T12:26:47.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-15T12:26:47.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T12:26:47.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T12:26:47.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T12:26:47.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-15T12:26:47.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T12:26:48.245 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T12:26:48.245 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:26:48.245 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:26:48.245 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:26:48.245 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:26:48.245 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:26:48.245 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:26:48.246 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:26:48.247 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:26:48.309 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-05-15T12:26:48.309 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:26:48.309 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:26:48.309 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:26:48.309 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:26:48.309 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:26:48.310 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:26:48.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-05-15T12:26:48.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-mon.a.asok log flush 2024-05-15T12:26:48.412 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:26:48.430 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=lrc 2024-05-15T12:26:48.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-15T12:26:48.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:26:48.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T12:26:48.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:26:48.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-15T12:26:48.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-15T12:26:48.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:26:48.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:26:48.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:26:48.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-15T12:26:48.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-15T12:26:48.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:26:48.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:26:48.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-15T12:26:48.432 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:26:48.432 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:26:48.432 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:26:48.432 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:26:48.432 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:26:48.433 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:26:48.433 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:26:48.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:26:48.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:26:48.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:26:48.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:26:48.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-15T12:26:48.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-15T12:26:48.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:26:48.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:26:48.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:26:48.434 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:26:48.434 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:26:48.434 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=lrc 2024-05-15T12:26:48.434 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-15T12:26:48.434 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:26:48.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=71ffa9ae-5b70-466d-892b-277268ae16db 2024-05-15T12:26:48.435 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 71ffa9ae-5b70-466d-892b-277268ae16db 2024-05-15T12:26:48.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 71ffa9ae-5b70-466d-892b-277268ae16db' 2024-05-15T12:26:48.435 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:26:48.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAIqkRmKI29GxAAtqBr5XC3m6d8yqm81VxTyQ== 2024-05-15T12:26:48.455 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAIqkRmKI29GxAAtqBr5XC3m6d8yqm81VxTyQ=="}' 2024-05-15T12:26:48.455 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 71ffa9ae-5b70-466d-892b-277268ae16db -i td/test-erasure-code-plugins/0/new.json 2024-05-15T12:26:48.772 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:26:48.788 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-15T12:26:48.789 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc --mkfs --key AQAIqkRmKI29GxAAtqBr5XC3m6d8yqm81VxTyQ== --osd-uuid 71ffa9ae-5b70-466d-892b-277268ae16db 2024-05-15T12:26:48.828 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:48.838+0000 7f6e2bf597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:26:48.850 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:48.858+0000 7f6e2bf597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:26:48.851 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:48.858+0000 7f6e2bf597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:26:48.852 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:48.862+0000 7f6e2bf597c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:26:48.852 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:48.862+0000 7f6e2bf597c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:26:48.852 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:48.862+0000 7f6e2bf597c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:26:48.881 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:48.890+0000 7f6e2bf597c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-15T12:26:55.274 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-15T12:26:55.274 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:26:55.275 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T12:26:55.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T12:26:55.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:26:55.835 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T12:26:55.836 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T12:26:55.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc 2024-05-15T12:26:55.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:26:55.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:26:55.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:26:55.866 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:55.873+0000 7f1c3c5767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:26:55.882 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:55.889+0000 7f1c3c5767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:26:55.883 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:55.889+0000 7f1c3c5767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:26:56.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T12:26:56.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:26:56.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T12:26:56.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:26:56.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:26:56.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:56.264 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:26:56.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:26:56.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:56.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:56.440 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:56.449+0000 7f1c3c5767c0 -1 Falling back to public interface 2024-05-15T12:26:56.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:57.697 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:57.697 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:57.697 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:26:57.698 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:26:57.698 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:57.698 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:58.122 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:26:59.005 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:26:59.013+0000 7f1c3c5767c0 -1 osd.0 0 log_to_monitors true 2024-05-15T12:26:59.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:26:59.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:26:59.125 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:26:59.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:26:59.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:26:59.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:26:59.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:00.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:00.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:00.636 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:27:00.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:27:00.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:00.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:01.070 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:02.072 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:02.072 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:02.072 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:27:02.073 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:27:02.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:02.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:02.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:03.596 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:03.596 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:03.596 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:27:03.597 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:27:03.597 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:03.597 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:04.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:05.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:05.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:05.064 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:27:05.064 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:27:05.064 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:05.064 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:05.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:06.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:06.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:06.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:27:06.496 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:27:06.497 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:06.497 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:07.030 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:08.031 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:08.032 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:08.032 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:27:08.032 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:27:08.032 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:08.033 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:08.460 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:09.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:09.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:09.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:27:09.462 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:27:09.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:09.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:09.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:10.997 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:10.997 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:10.997 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:27:10.997 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:27:10.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:10.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:11.441 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:12.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:12.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:12.444 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:27:12.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:27:12.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:12.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:13.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:14.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:14.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:14.427 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:27:14.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:27:14.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:14.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:15.130 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:16.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:16.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:16.133 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:27:16.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:27:16.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:16.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:17.168 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:18.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:18.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:18.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:27:18.171 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:27:18.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:18.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:18.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:20.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:20.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:20.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:27:20.002 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:27:20.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:20.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:20.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:21.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:21.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:21.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:27:21.821 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:27:21.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:21.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:22.544 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:23.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:23.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:23.546 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:27:23.547 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:27:23.547 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:23.547 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:23.969 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4136782779,v1:127.0.0.1:6803/4136782779] [v2:127.0.0.1:6804/4136782779,v1:127.0.0.1:6805/4136782779] exists,up 71ffa9ae-5b70-466d-892b-277268ae16db 2024-05-15T12:27:23.971 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:27:23.971 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:27:23.971 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:27:23.971 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-05-15T12:27:23.971 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T12:27:23.971 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T12:27:23.972 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:27:23.972 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:27:23.972 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:27:23.972 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-osd.0.asok 2024-05-15T12:27:23.972 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-05-15T12:27:23.972 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-osd.0.asok log flush 2024-05-15T12:27:24.082 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:27:24.094 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-05-15T12:27:24.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-05-15T12:27:24.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-05-15T12:27:24.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:27:24.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:27:24.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:27:24.101 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:27:24.101 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:27:24.101 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:27:24.101 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:27:24.101 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:27:24.209 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:27:24.209 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:27:24.210 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:27:24.210 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:27:24.211 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:27:24.212 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:27:24.212 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:27:24.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:27:24.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:27:24.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:27:24.213 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:27:24.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:27:24.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:27:24.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:27:24.222 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:27:24.222 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:27:24.222 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:27:24.222 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:27:24.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:27:24.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:27:24.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-05-15T12:27:24.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-05-15T12:27:24.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-15T12:27:24.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-15T12:27:24.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:27:24.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:27:24.225 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:27:24.225 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:27:24.225 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:27:24.225 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:27:24.225 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:27:24.225 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:27:24.225 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:27:24.226 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:27:24.226 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:27:24.227 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:27:24.227 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:27:24.227 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:27:24.227 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:27:24.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:27:24.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:27:24.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:27:24.229 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:27:24.230 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:27:24.230 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:27:24.231 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:27:24.232 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:27:24.232 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:27:24.232 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:27:24.232 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:27:24.232 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:27:24.232 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:27:24.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-15T12:27:24.234 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T12:27:24.234 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:27:24.234 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:27:24.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20204 2024-05-15T12:27:24.236 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T12:27:24.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-15T12:27:24.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T12:27:24.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-15T12:27:24.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=isa 2024-05-15T12:27:24.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-15T12:27:24.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T12:27:24.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T12:27:24.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T12:27:24.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-15T12:27:24.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=isa 2024-05-15T12:27:24.645 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T12:27:24.645 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:27:24.645 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:27:24.646 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:27:24.646 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:27:24.646 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:27:24.646 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:27:24.646 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=isa 2024-05-15T12:27:24.857 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T12:27:24.906 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T12:27:24.906 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:27:24.906 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:27:24.906 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T12:27:24.906 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T12:27:24.906 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:27:24.906 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:27:24.907 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:27:24.907 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:27:24.907 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:27:24.907 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:27:24.907 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:27:24.907 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:27:24.907 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get fsid 2024-05-15T12:27:24.980 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T12:27:24.980 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:27:24.980 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:27:24.980 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T12:27:24.980 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T12:27:24.980 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:27:24.980 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:27:24.980 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:27:24.981 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:27:24.981 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:27:24.981 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:27:24.981 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:27:24.981 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:27:24.981 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get mon_host 2024-05-15T12:27:25.094 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-05-15T12:27:25.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-15T12:27:25.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T12:27:25.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T12:27:25.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T12:27:25.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-15T12:27:25.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T12:27:25.997 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T12:27:25.997 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:27:25.997 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:27:25.997 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:27:25.997 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:27:25.998 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:27:25.998 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:27:25.998 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:27:25.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:27:26.035 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-05-15T12:27:26.035 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:27:26.035 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:27:26.035 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:27:26.036 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:27:26.036 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:27:26.036 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:27:26.036 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-05-15T12:27:26.036 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-mon.a.asok log flush 2024-05-15T12:27:26.135 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:27:26.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=isa 2024-05-15T12:27:26.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-15T12:27:26.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:27:26.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T12:27:26.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:27:26.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-15T12:27:26.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-15T12:27:26.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:27:26.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:27:26.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:27:26.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-15T12:27:26.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-15T12:27:26.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:27:26.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:27:26.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-15T12:27:26.150 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:27:26.151 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:27:26.151 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:27:26.153 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:27:26.153 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:27:26.153 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:27:26.153 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:27:26.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:27:26.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:27:26.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:27:26.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:27:26.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-15T12:27:26.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-15T12:27:26.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:27:26.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:27:26.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:27:26.154 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:27:26.154 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:27:26.154 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=isa 2024-05-15T12:27:26.154 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-15T12:27:26.154 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:27:26.154 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 8ade668f-74eb-4037-9694-c13610a5027c 2024-05-15T12:27:26.154 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8ade668f-74eb-4037-9694-c13610a5027c 2024-05-15T12:27:26.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8ade668f-74eb-4037-9694-c13610a5027c' 2024-05-15T12:27:26.155 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:27:26.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAuqkRmF7nGChAAoW9BFQQtj8UUTeacjO9qtQ== 2024-05-15T12:27:26.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAuqkRmF7nGChAAoW9BFQQtj8UUTeacjO9qtQ=="}' 2024-05-15T12:27:26.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8ade668f-74eb-4037-9694-c13610a5027c -i td/test-erasure-code-plugins/0/new.json 2024-05-15T12:27:26.460 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:27:26.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-15T12:27:26.478 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa --mkfs --key AQAuqkRmF7nGChAAoW9BFQQtj8UUTeacjO9qtQ== --osd-uuid 8ade668f-74eb-4037-9694-c13610a5027c 2024-05-15T12:27:26.513 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:27:26.521+0000 7ff1e0cfe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:27:26.528 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:27:26.537+0000 7ff1e0cfe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:27:26.529 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:27:26.537+0000 7ff1e0cfe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:27:26.530 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:27:26.537+0000 7ff1e0cfe7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:27:26.530 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:27:26.537+0000 7ff1e0cfe7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:27:26.530 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:27:26.537+0000 7ff1e0cfe7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:27:26.569 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:27:26.577+0000 7ff1e0cfe7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-15T12:27:32.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-15T12:27:32.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:27:32.923 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T12:27:32.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T12:27:32.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:27:33.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T12:27:33.504 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T12:27:33.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:27:33.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa 2024-05-15T12:27:33.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:27:33.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:27:33.534 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:27:33.541+0000 7f016bee87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:27:33.558 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:27:33.565+0000 7f016bee87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:27:33.561 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:27:33.569+0000 7f016bee87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:27:33.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T12:27:33.929 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:27:33.929 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T12:27:33.929 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:27:33.929 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:27:33.929 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:33.929 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:27:33.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:27:33.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:33.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:34.125 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:27:34.133+0000 7f016bee87c0 -1 Falling back to public interface 2024-05-15T12:27:34.342 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:35.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:35.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:35.344 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:27:35.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:27:35.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:35.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:35.759 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:36.699 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:27:36.705+0000 7f016bee87c0 -1 osd.0 0 log_to_monitors true 2024-05-15T12:27:36.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:36.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:36.760 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:27:36.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:27:36.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:36.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:37.418 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:38.420 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:38.420 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:38.420 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:27:38.420 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:27:38.421 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:38.421 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:38.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:39.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:39.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:39.881 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:27:39.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:27:39.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:39.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:40.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:41.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:41.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:41.337 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:27:41.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:27:41.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:41.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:41.848 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:42.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:42.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:42.850 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:27:42.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:27:42.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:42.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:43.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:44.284 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:44.284 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:44.285 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:27:44.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:27:44.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:44.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:44.783 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:45.784 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:45.784 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:45.785 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:27:45.785 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:27:45.785 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:45.785 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:46.218 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:47.220 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:47.220 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:47.220 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:27:47.220 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:27:47.221 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:47.221 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:47.653 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:48.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:48.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:48.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:27:48.655 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:27:48.656 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:48.656 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:49.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:50.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:50.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:50.084 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:27:50.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:27:50.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:50.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:50.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:51.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:51.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:51.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:27:51.518 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:27:51.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:51.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:52.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:53.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:53.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:53.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:27:53.172 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:27:53.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:53.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:53.857 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:54.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:54.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:54.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:27:54.859 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:27:54.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:54.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:55.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:56.349 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:56.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:56.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:27:56.350 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:27:56.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:56.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:57.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:27:58.137 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:27:58.137 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:27:58.137 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:27:58.138 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:27:58.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:27:58.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:27:58.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:00.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:00.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:00.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:28:00.002 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:28:00.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:00.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:00.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:01.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:01.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:01.587 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T12:28:01.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T12:28:01.588 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:01.588 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:02.037 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3096580200,v1:127.0.0.1:6803/3096580200] [v2:127.0.0.1:6804/3096580200,v1:127.0.0.1:6805/3096580200] exists,up 8ade668f-74eb-4037-9694-c13610a5027c 2024-05-15T12:28:02.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:28:02.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:28:02.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:28:02.039 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-05-15T12:28:02.039 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T12:28:02.039 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T12:28:02.039 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:28:02.039 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:28:02.039 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:28:02.040 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-osd.0.asok 2024-05-15T12:28:02.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-05-15T12:28:02.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-osd.0.asok log flush 2024-05-15T12:28:02.154 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:28:02.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-05-15T12:28:02.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-05-15T12:28:02.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-05-15T12:28:02.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:28:02.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:28:02.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:28:02.176 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:28:02.176 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:28:02.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:28:02.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:28:02.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:28:02.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:28:02.486 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:28:02.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:28:02.488 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:28:02.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:28:02.490 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:28:02.490 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:28:02.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:28:02.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:28:02.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:28:02.492 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:28:02.493 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:28:02.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:28:02.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:28:02.503 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:28:02.503 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:28:02.503 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:28:02.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:28:02.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:28:02.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:28:02.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:72: TEST_preload_no_warning: return 0 2024-05-15T12:28:02.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-05-15T12:28:02.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning_default td/test-erasure-code-plugins 2024-05-15T12:28:02.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:76: TEST_preload_no_warning_default: local dir=td/test-erasure-code-plugins 2024-05-15T12:28:02.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:78: TEST_preload_no_warning_default: setup td/test-erasure-code-plugins 2024-05-15T12:28:02.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-15T12:28:02.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-15T12:28:02.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:28:02.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:28:02.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:28:02.506 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:28:02.506 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:28:02.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:28:02.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:28:02.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:28:02.508 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:28:02.508 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:28:02.509 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:28:02.510 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:28:02.511 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:28:02.511 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:28:02.511 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:28:02.516 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:28:02.516 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:28:02.516 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:28:02.517 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:28:02.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:28:02.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:28:02.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:28:02.517 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:28:02.517 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:28:02.518 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:28:02.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:28:02.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:28:02.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:28:02.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-15T12:28:02.521 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T12:28:02.521 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:28:02.521 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:28:02.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20204 2024-05-15T12:28:02.523 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T12:28:02.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-15T12:28:02.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T12:28:02.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-15T12:28:02.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:79: TEST_preload_no_warning_default: run_mon td/test-erasure-code-plugins a 2024-05-15T12:28:02.524 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-15T12:28:02.524 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T12:28:02.524 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T12:28:02.524 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T12:28:02.524 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-15T12:28:02.524 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2024-05-15T12:28:02.940 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T12:28:02.941 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:28:02.941 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:28:02.941 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:28:02.942 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:28:02.942 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:28:02.942 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:28:02.942 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-15T12:28:03.168 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T12:28:03.169 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T12:28:03.169 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:28:03.169 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:28:03.169 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T12:28:03.169 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T12:28:03.170 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:28:03.170 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:28:03.170 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:28:03.170 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:28:03.171 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:28:03.171 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:28:03.171 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:28:03.171 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:28:03.171 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get fsid 2024-05-15T12:28:03.288 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T12:28:03.288 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:28:03.288 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:28:03.288 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T12:28:03.288 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T12:28:03.289 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:28:03.289 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:28:03.289 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:28:03.289 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:28:03.290 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:28:03.290 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:28:03.290 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:28:03.290 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:28:03.290 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get mon_host 2024-05-15T12:28:03.408 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: get_asok_path mon.a 2024-05-15T12:28:03.409 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:28:03.409 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:28:03.409 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:28:03.409 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:28:03.409 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:28:03.410 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:28:03.410 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: CEPH_ARGS= 2024-05-15T12:28:03.410 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-mon.a.asok log flush 2024-05-15T12:28:03.507 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:28:03.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:81: TEST_preload_no_warning_default: run_mgr td/test-erasure-code-plugins x 2024-05-15T12:28:03.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-15T12:28:03.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T12:28:03.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T12:28:03.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T12:28:03.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-15T12:28:03.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T12:28:04.435 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T12:28:04.436 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:28:04.436 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:28:04.436 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:28:04.436 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:28:04.436 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:28:04.436 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:28:04.436 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:28:04.437 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:28:04.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:82: TEST_preload_no_warning_default: run_osd td/test-erasure-code-plugins 0 2024-05-15T12:28:04.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-15T12:28:04.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:28:04.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T12:28:04.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:28:04.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-15T12:28:04.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-15T12:28:04.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:28:04.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:28:04.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:28:04.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-15T12:28:04.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-15T12:28:04.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:28:04.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:28:04.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-15T12:28:04.503 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:28:04.503 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:28:04.503 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:28:04.503 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:28:04.503 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:28:04.503 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:28:04.503 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:28:04.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:28:04.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:28:04.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:28:04.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:28:04.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-15T12:28:04.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-15T12:28:04.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:28:04.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:28:04.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:28:04.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:28:04.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:28:04.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:28:04.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-15T12:28:04.505 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:28:04.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c1ef1b2f-4de7-481e-908b-9ebce6199469 2024-05-15T12:28:04.505 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 c1ef1b2f-4de7-481e-908b-9ebce6199469 2024-05-15T12:28:04.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c1ef1b2f-4de7-481e-908b-9ebce6199469' 2024-05-15T12:28:04.506 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:28:04.524 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBUqkRmeJ3xHxAAgD/nkt2jcKZC9HcTjbGSKA== 2024-05-15T12:28:04.524 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBUqkRmeJ3xHxAAgD/nkt2jcKZC9HcTjbGSKA=="}' 2024-05-15T12:28:04.524 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c1ef1b2f-4de7-481e-908b-9ebce6199469 -i td/test-erasure-code-plugins/0/new.json 2024-05-15T12:28:04.814 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:28:04.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-15T12:28:04.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBUqkRmeJ3xHxAAgD/nkt2jcKZC9HcTjbGSKA== --osd-uuid c1ef1b2f-4de7-481e-908b-9ebce6199469 2024-05-15T12:28:04.862 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:04.873+0000 7f11500797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:28:04.874 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:04.881+0000 7f11500797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:28:04.875 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:04.885+0000 7f11500797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:28:04.876 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:04.885+0000 7f11500797c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:28:04.876 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:04.885+0000 7f11500797c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:28:04.876 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:04.885+0000 7f11500797c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:28:04.914 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:04.921+0000 7f11500797c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-15T12:28:11.294 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-15T12:28:11.294 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:28:11.295 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T12:28:11.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T12:28:11.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:28:11.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T12:28:11.863 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T12:28:11.864 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:28:11.864 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:28:11.864 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:28:11.864 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:28:11.896 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:11.905+0000 7f4eea3147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:28:11.898 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:11.905+0000 7f4eea3147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:28:11.899 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:11.909+0000 7f4eea3147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:28:12.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T12:28:12.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:28:12.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T12:28:12.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:28:12.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:28:12.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:12.345 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:28:12.345 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:28:12.345 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:12.345 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:12.467 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:12.477+0000 7f4eea3147c0 -1 Falling back to public interface 2024-05-15T12:28:12.818 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:13.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:13.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:13.820 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:28:13.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:28:13.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:13.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:14.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:15.064 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:15.073+0000 7f4eea3147c0 -1 osd.0 0 log_to_monitors true 2024-05-15T12:28:15.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:15.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:15.236 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:28:15.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:28:15.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:15.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:15.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:16.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:16.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:16.747 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:28:16.747 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:28:16.747 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:16.747 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:17.192 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:18.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:18.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:18.194 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:28:18.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:28:18.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:18.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:18.657 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:19.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:19.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:19.659 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:28:19.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:28:19.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:19.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:20.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:21.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:21.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:21.269 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:28:21.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:28:21.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:21.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:21.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:22.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:22.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:22.707 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:28:22.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:28:22.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:22.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:23.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:24.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:24.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:24.182 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:28:24.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:28:24.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:24.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:24.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:25.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:25.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:25.633 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:28:25.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:28:25.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:25.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:26.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:27.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:27.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:27.087 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:28:27.088 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:28:27.088 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:27.088 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:27.512 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:28.514 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:28.514 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:28.514 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:28:28.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:28:28.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:28.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:28.929 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:29.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:29.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:29.931 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:28:29.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:28:29.932 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:29.932 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:30.598 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:31.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:31.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:31.601 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:28:31.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:28:31.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:31.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:32.321 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:33.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:33.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:33.324 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:28:33.324 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:28:33.324 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:33.324 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:34.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:35.277 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:35.277 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:35.277 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:28:35.278 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:28:35.278 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:35.278 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:36.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:37.157 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:37.157 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:37.158 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:28:37.158 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:28:37.158 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:37.158 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:37.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:38.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:38.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:38.765 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:28:38.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:28:38.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:38.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:39.270 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1700752501,v1:127.0.0.1:6803/1700752501] [v2:127.0.0.1:6804/1700752501,v1:127.0.0.1:6805/1700752501] exists,up c1ef1b2f-4de7-481e-908b-9ebce6199469 2024-05-15T12:28:39.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:28:39.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:28:39.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:28:39.271 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: get_asok_path osd.0 2024-05-15T12:28:39.271 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T12:28:39.271 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T12:28:39.272 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:28:39.272 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:28:39.272 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:28:39.272 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-osd.0.asok 2024-05-15T12:28:39.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: CEPH_ARGS= 2024-05-15T12:28:39.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-osd.0.asok log flush 2024-05-15T12:28:39.388 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:28:39.401 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:84: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/mon.a.log 2024-05-15T12:28:39.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:85: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/osd.0.log 2024-05-15T12:28:39.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:86: TEST_preload_no_warning_default: teardown td/test-erasure-code-plugins 2024-05-15T12:28:39.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:28:39.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:28:39.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:28:39.409 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:28:39.409 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:28:39.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:28:39.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:28:39.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:28:39.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:28:39.718 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:28:39.719 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:28:39.719 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:28:39.721 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:28:39.721 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:28:39.722 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:28:39.723 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:28:39.723 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:28:39.724 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:28:39.724 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:28:39.725 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:28:39.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:28:39.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:28:39.735 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:28:39.735 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:28:39.735 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:28:39.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:28:39.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:28:39.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:28:39.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:88: TEST_preload_no_warning_default: return 0 2024-05-15T12:28:39.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-05-15T12:28:39.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_warning td/test-erasure-code-plugins 2024-05-15T12:28:39.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:41: TEST_preload_warning: local dir=td/test-erasure-code-plugins 2024-05-15T12:28:39.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-05-15T12:28:39.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-05-15T12:28:39.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-15T12:28:39.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-15T12:28:39.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:28:39.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:28:39.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:28:39.739 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:28:39.739 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:28:39.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:28:39.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:28:39.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:28:39.740 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:28:39.741 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:28:39.742 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:28:39.742 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:28:39.743 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:28:39.743 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:28:39.744 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:28:39.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:28:39.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:28:39.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:28:39.746 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:28:39.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:28:39.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:28:39.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:28:39.750 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:28:39.750 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:28:39.750 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:28:39.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:28:39.751 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:28:39.751 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:28:39.751 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-15T12:28:39.753 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T12:28:39.753 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:28:39.753 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:28:39.753 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20204 2024-05-15T12:28:39.754 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T12:28:39.755 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-15T12:28:39.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T12:28:39.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-15T12:28:39.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_generic 2024-05-15T12:28:39.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-15T12:28:39.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T12:28:39.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T12:28:39.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T12:28:39.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-15T12:28:39.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_generic 2024-05-15T12:28:40.153 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T12:28:40.153 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:28:40.153 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:28:40.154 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:28:40.154 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:28:40.154 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:28:40.154 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:28:40.154 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_generic 2024-05-15T12:28:40.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T12:28:40.365 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T12:28:40.366 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:28:40.366 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:28:40.366 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T12:28:40.366 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T12:28:40.367 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:28:40.367 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:28:40.367 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:28:40.368 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:28:40.368 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:28:40.368 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:28:40.368 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:28:40.368 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:28:40.368 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get fsid 2024-05-15T12:28:40.487 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T12:28:40.487 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:28:40.487 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:28:40.487 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T12:28:40.487 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T12:28:40.488 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:28:40.488 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:28:40.488 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:28:40.488 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:28:40.488 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:28:40.488 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:28:40.489 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:28:40.489 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:28:40.489 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get mon_host 2024-05-15T12:28:40.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-05-15T12:28:40.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-15T12:28:40.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T12:28:40.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T12:28:40.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T12:28:40.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-15T12:28:40.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T12:28:41.505 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T12:28:41.505 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:28:41.505 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:28:41.506 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:28:41.506 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:28:41.506 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:28:41.506 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:28:41.506 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:28:41.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:28:41.543 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-05-15T12:28:41.543 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:28:41.543 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:28:41.543 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:28:41.543 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:28:41.543 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:28:41.543 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:28:41.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-05-15T12:28:41.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-mon.a.asok log flush 2024-05-15T12:28:41.643 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:28:41.657 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_generic 2024-05-15T12:28:41.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-15T12:28:41.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:28:41.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T12:28:41.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:28:41.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-15T12:28:41.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-15T12:28:41.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:28:41.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:28:41.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:28:41.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-15T12:28:41.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-15T12:28:41.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:28:41.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:28:41.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-15T12:28:41.659 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:28:41.659 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:28:41.659 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:28:41.660 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:28:41.660 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:28:41.660 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:28:41.660 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:28:41.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:28:41.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:28:41.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:28:41.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:28:41.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-15T12:28:41.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-15T12:28:41.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:28:41.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:28:41.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:28:41.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:28:41.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:28:41.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_generic 2024-05-15T12:28:41.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-15T12:28:41.662 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:28:41.662 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0acb4397-61ec-4784-a990-e5a4ff955286 2024-05-15T12:28:41.662 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 0acb4397-61ec-4784-a990-e5a4ff955286 2024-05-15T12:28:41.662 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0acb4397-61ec-4784-a990-e5a4ff955286' 2024-05-15T12:28:41.662 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:28:41.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB5qkRm49qJKRAAf5BzdOMLPa2fwgngQw9vgA== 2024-05-15T12:28:41.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB5qkRm49qJKRAAf5BzdOMLPa2fwgngQw9vgA=="}' 2024-05-15T12:28:41.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0acb4397-61ec-4784-a990-e5a4ff955286 -i td/test-erasure-code-plugins/0/new.json 2024-05-15T12:28:41.985 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:28:41.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-15T12:28:42.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic --mkfs --key AQB5qkRm49qJKRAAf5BzdOMLPa2fwgngQw9vgA== --osd-uuid 0acb4397-61ec-4784-a990-e5a4ff955286 2024-05-15T12:28:42.039 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:42.048+0000 7f897f7f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:28:42.053 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:42.060+0000 7f897f7f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:28:42.054 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:42.064+0000 7f897f7f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:28:42.055 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:42.064+0000 7f897f7f17c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:28:42.055 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:42.064+0000 7f897f7f17c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:28:42.055 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:42.064+0000 7f897f7f17c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:28:42.093 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:42.100+0000 7f897f7f17c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-15T12:28:48.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-15T12:28:48.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:28:48.383 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T12:28:48.383 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T12:28:48.384 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:28:48.975 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T12:28:48.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T12:28:48.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic 2024-05-15T12:28:48.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:28:48.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:28:48.977 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:28:49.007 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:49.016+0000 7f812d0d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:28:49.009 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:49.016+0000 7f812d0d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:28:49.010 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:49.020+0000 7f812d0d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:28:49.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T12:28:49.437 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:28:49.437 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T12:28:49.437 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:28:49.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:28:49.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:28:49.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:49.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:28:49.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:49.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:49.584 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:49.592+0000 7f812d0d17c0 -1 Falling back to public interface 2024-05-15T12:28:49.929 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:50.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:50.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:50.931 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:28:50.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:28:50.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:50.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:51.345 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:52.131 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:28:52.140+0000 7f812d0d17c0 -1 osd.0 0 log_to_monitors true 2024-05-15T12:28:52.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:52.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:52.347 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:28:52.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:28:52.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:52.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:52.973 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:53.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:53.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:53.975 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:28:53.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:28:53.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:53.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:54.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:55.453 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:55.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:55.454 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:28:55.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:28:55.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:55.455 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:55.974 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:56.978 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:56.978 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:56.978 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:28:56.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:28:56.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:56.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:57.413 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:58.415 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:58.415 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:58.415 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:28:58.415 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:28:58.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:58.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:28:58.996 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:28:59.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:28:59.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:28:59.998 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:28:59.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:28:59.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:28:59.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:00.460 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:01.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:01.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:01.462 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:29:01.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:29:01.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:01.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:01.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:02.972 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:02.972 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:02.972 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:29:02.973 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:29:02.973 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:02.973 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:03.437 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:04.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:04.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:04.439 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:29:04.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:29:04.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:04.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:05.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:06.341 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:06.341 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:06.341 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:29:06.342 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:29:06.342 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:06.342 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:07.437 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:08.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:08.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:08.439 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:29:08.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:29:08.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:08.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:09.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:10.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:10.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:10.008 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:29:10.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:29:10.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:10.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:10.801 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:11.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:11.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:11.803 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:29:11.804 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:29:11.804 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:11.804 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:12.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:13.714 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:13.714 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:13.714 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:29:13.715 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:29:13.715 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:13.715 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:14.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:15.493 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:15.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:15.494 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:29:15.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:29:15.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:15.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:15.978 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1798718195,v1:127.0.0.1:6803/1798718195] [v2:127.0.0.1:6804/1798718195,v1:127.0.0.1:6805/1798718195] exists,up 0acb4397-61ec-4784-a990-e5a4ff955286 2024-05-15T12:29:15.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:29:15.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:29:15.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:29:15.979 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-05-15T12:29:15.979 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T12:29:15.979 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T12:29:15.980 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:29:15.980 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:29:15.980 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:29:15.980 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-osd.0.asok 2024-05-15T12:29:15.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-05-15T12:29:15.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-osd.0.asok log flush 2024-05-15T12:29:16.105 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:29:16.120 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2024-05-15T12:29:16.125 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:28:40.200+0000 7f8524886c80 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-05-15T12:29:16.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/osd.0.log 2024-05-15T12:29:16.127 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:28:49.596+0000 7f812d0d17c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-05-15T12:29:16.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-05-15T12:29:16.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:29:16.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:29:16.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:29:16.128 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:29:16.129 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:29:16.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:29:16.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:29:16.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:29:16.238 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:29:16.238 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:29:16.238 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:29:16.239 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:29:16.240 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:29:16.240 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:29:16.240 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:29:16.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:29:16.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:29:16.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:29:16.241 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:29:16.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:29:16.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:29:16.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:29:16.249 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:29:16.249 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:29:16.249 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:29:16.249 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:29:16.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:29:16.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:29:16.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-05-15T12:29:16.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-05-15T12:29:16.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-15T12:29:16.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-15T12:29:16.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:29:16.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:29:16.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:29:16.251 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:29:16.251 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:29:16.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:29:16.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:29:16.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:29:16.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:29:16.253 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:29:16.253 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:29:16.254 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:29:16.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:29:16.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:29:16.255 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:29:16.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:29:16.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:29:16.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:29:16.256 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:29:16.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:29:16.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:29:16.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:29:16.259 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:29:16.259 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:29:16.259 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:29:16.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:29:16.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:29:16.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:29:16.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-15T12:29:16.262 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T12:29:16.262 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:29:16.262 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:29:16.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20204 2024-05-15T12:29:16.264 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T12:29:16.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-15T12:29:16.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T12:29:16.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-15T12:29:16.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse3 2024-05-15T12:29:16.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-15T12:29:16.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T12:29:16.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T12:29:16.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T12:29:16.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-15T12:29:16.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse3 2024-05-15T12:29:16.716 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T12:29:16.716 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:29:16.717 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:29:16.717 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:29:16.717 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:29:16.717 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:29:16.717 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:29:16.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse3 2024-05-15T12:29:16.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T12:29:16.920 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T12:29:16.920 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:29:16.921 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:29:16.921 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T12:29:16.921 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T12:29:16.921 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:29:16.921 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:29:16.921 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:29:16.922 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:29:16.922 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:29:16.922 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:29:16.922 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:29:16.923 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:29:16.923 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get fsid 2024-05-15T12:29:17.040 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T12:29:17.040 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:29:17.040 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:29:17.040 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T12:29:17.040 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T12:29:17.041 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:29:17.041 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:29:17.041 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:29:17.041 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:29:17.041 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:29:17.042 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:29:17.042 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:29:17.042 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:29:17.042 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get mon_host 2024-05-15T12:29:17.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-05-15T12:29:17.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-15T12:29:17.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T12:29:17.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T12:29:17.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T12:29:17.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-15T12:29:17.145 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T12:29:18.060 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T12:29:18.060 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:29:18.060 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:29:18.061 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:29:18.061 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:29:18.061 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:29:18.061 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:29:18.062 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:29:18.064 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:29:18.096 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-05-15T12:29:18.096 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:29:18.096 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:29:18.096 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:29:18.096 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:29:18.096 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:29:18.096 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:29:18.096 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-05-15T12:29:18.096 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-mon.a.asok log flush 2024-05-15T12:29:18.197 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:29:18.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse3 2024-05-15T12:29:18.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-15T12:29:18.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:29:18.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T12:29:18.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:29:18.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-15T12:29:18.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-15T12:29:18.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:29:18.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:29:18.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:29:18.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-15T12:29:18.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-15T12:29:18.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:29:18.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:29:18.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-15T12:29:18.215 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:29:18.215 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:29:18.215 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:29:18.216 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:29:18.216 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:29:18.216 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:29:18.216 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:29:18.216 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:29:18.216 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:29:18.216 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:29:18.216 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:29:18.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-15T12:29:18.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-15T12:29:18.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:29:18.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:29:18.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:29:18.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:29:18.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:29:18.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse3 2024-05-15T12:29:18.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-15T12:29:18.217 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:29:18.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b9b5b0ea-7a9a-4c97-a1d1-2096514c9567 2024-05-15T12:29:18.218 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 b9b5b0ea-7a9a-4c97-a1d1-2096514c9567 2024-05-15T12:29:18.218 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b9b5b0ea-7a9a-4c97-a1d1-2096514c9567' 2024-05-15T12:29:18.218 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:29:18.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCeqkRmOLe5DhAAKhp/T2duShlzjLpj5hnjcQ== 2024-05-15T12:29:18.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCeqkRmOLe5DhAAKhp/T2duShlzjLpj5hnjcQ=="}' 2024-05-15T12:29:18.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b9b5b0ea-7a9a-4c97-a1d1-2096514c9567 -i td/test-erasure-code-plugins/0/new.json 2024-05-15T12:29:18.539 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:29:18.556 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-15T12:29:18.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 --mkfs --key AQCeqkRmOLe5DhAAKhp/T2duShlzjLpj5hnjcQ== --osd-uuid b9b5b0ea-7a9a-4c97-a1d1-2096514c9567 2024-05-15T12:29:18.587 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:29:18.596+0000 7f01b43437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:29:18.599 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:29:18.608+0000 7f01b43437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:29:18.601 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:29:18.608+0000 7f01b43437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:29:18.601 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:29:18.608+0000 7f01b43437c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:29:18.601 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:29:18.608+0000 7f01b43437c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:29:18.601 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:29:18.608+0000 7f01b43437c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:29:18.640 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:29:18.648+0000 7f01b43437c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-15T12:29:25.154 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-15T12:29:25.154 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:29:25.155 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T12:29:25.156 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T12:29:25.156 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:29:25.718 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T12:29:25.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T12:29:25.719 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:29:25.719 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:29:25.719 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:29:25.719 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 2024-05-15T12:29:25.752 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:29:25.760+0000 7f6a83fd07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:29:25.771 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:29:25.780+0000 7f6a83fd07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:29:25.772 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:29:25.780+0000 7f6a83fd07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:29:26.161 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T12:29:26.161 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:29:26.161 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T12:29:26.161 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:29:26.161 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:29:26.161 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:26.162 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:29:26.162 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:29:26.162 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:26.162 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:26.338 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:29:26.348+0000 7f6a83fd07c0 -1 Falling back to public interface 2024-05-15T12:29:26.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:27.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:27.595 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:27.595 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:29:27.595 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:29:27.595 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:27.595 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:28.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:28.913 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:29:28.920+0000 7f6a83fd07c0 -1 osd.0 0 log_to_monitors true 2024-05-15T12:29:29.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:29.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:29.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:29:29.014 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:29:29.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:29.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:29.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:30.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:30.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:30.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:29:30.587 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:29:30.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:30.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:31.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:32.021 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:32.022 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:32.022 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:29:32.022 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:29:32.022 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:32.022 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:32.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:33.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:33.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:33.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:29:33.504 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:29:33.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:33.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:33.987 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:34.990 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:34.990 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:34.990 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:29:34.990 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:29:34.991 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:34.991 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:35.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:36.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:36.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:36.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:29:36.464 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:29:36.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:36.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:36.914 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:37.915 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:37.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:37.916 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:29:37.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:29:37.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:37.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:38.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:39.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:39.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:39.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:29:39.382 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:29:39.383 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:39.383 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:39.915 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:40.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:40.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:40.918 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:29:40.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:29:40.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:40.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:41.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:42.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:42.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:42.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:29:42.359 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:29:42.359 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:42.359 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:43.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:44.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:44.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:44.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:29:44.288 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:29:44.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:44.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:45.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:46.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:46.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:46.057 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:29:46.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:29:46.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:46.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:46.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:47.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:47.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:47.860 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:29:47.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:29:47.861 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:47.861 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:48.646 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:49.648 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:49.648 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:49.648 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:29:49.648 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:29:49.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:49.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:50.210 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:51.212 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:51.212 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:51.212 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:29:51.213 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:29:51.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:51.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:52.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:29:53.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:29:53.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:29:53.261 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:29:53.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:29:53.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:29:53.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:29:53.745 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/947194285,v1:127.0.0.1:6803/947194285] [v2:127.0.0.1:6804/947194285,v1:127.0.0.1:6805/947194285] exists,up b9b5b0ea-7a9a-4c97-a1d1-2096514c9567 2024-05-15T12:29:53.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:29:53.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:29:53.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:29:53.746 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-05-15T12:29:53.747 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T12:29:53.747 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T12:29:53.747 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:29:53.747 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:29:53.747 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:29:53.747 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-osd.0.asok 2024-05-15T12:29:53.747 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-05-15T12:29:53.747 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-osd.0.asok log flush 2024-05-15T12:29:53.862 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:29:53.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2024-05-15T12:29:53.880 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:29:16.764+0000 7fd1786f9c80 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-05-15T12:29:53.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/osd.0.log 2024-05-15T12:29:53.881 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:29:26.348+0000 7f6a83fd07c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-05-15T12:29:53.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-05-15T12:29:53.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:29:53.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:29:53.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:29:53.883 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:29:53.883 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:29:53.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:29:53.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:29:53.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:29:53.992 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:29:53.992 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:29:53.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:29:53.994 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:29:53.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:29:53.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:29:53.995 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:29:53.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:29:53.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:29:53.996 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:29:53.996 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:29:53.997 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:29:53.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:29:53.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:29:54.004 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:29:54.004 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:29:54.004 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:29:54.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:29:54.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:29:54.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:29:54.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-05-15T12:29:54.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-05-15T12:29:54.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-15T12:29:54.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-15T12:29:54.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:29:54.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:29:54.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:29:54.007 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:29:54.007 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:29:54.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:29:54.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:29:54.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:29:54.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:29:54.008 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:29:54.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:29:54.009 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:29:54.010 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:29:54.010 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:29:54.010 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:29:54.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:29:54.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:29:54.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:29:54.012 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:29:54.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:29:54.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:29:54.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:29:54.015 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:29:54.015 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:29:54.015 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:29:54.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:29:54.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:29:54.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:29:54.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-15T12:29:54.017 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T12:29:54.017 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:29:54.018 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:29:54.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20204 2024-05-15T12:29:54.020 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T12:29:54.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-15T12:29:54.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T12:29:54.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-15T12:29:54.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse4 2024-05-15T12:29:54.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-15T12:29:54.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T12:29:54.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T12:29:54.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T12:29:54.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-15T12:29:54.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse4 2024-05-15T12:29:54.410 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T12:29:54.411 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:29:54.411 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:29:54.411 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:29:54.412 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:29:54.412 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:29:54.412 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:29:54.412 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse4 2024-05-15T12:29:54.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T12:29:54.644 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T12:29:54.644 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:29:54.644 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:29:54.644 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T12:29:54.645 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T12:29:54.645 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:29:54.645 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:29:54.645 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:29:54.646 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:29:54.646 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:29:54.646 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:29:54.646 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:29:54.646 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:29:54.646 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get fsid 2024-05-15T12:29:54.765 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T12:29:54.765 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:29:54.765 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:29:54.766 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T12:29:54.766 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T12:29:54.766 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:29:54.766 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:29:54.766 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:29:54.766 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:29:54.766 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:29:54.767 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:29:54.767 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:29:54.767 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:29:54.767 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get mon_host 2024-05-15T12:29:54.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-05-15T12:29:54.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-15T12:29:54.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T12:29:54.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T12:29:54.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T12:29:54.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-15T12:29:54.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T12:29:55.804 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T12:29:55.804 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:29:55.804 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:29:55.804 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:29:55.804 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:29:55.804 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:29:55.805 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:29:55.805 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:29:55.806 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:29:55.838 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-05-15T12:29:55.838 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:29:55.838 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:29:55.838 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:29:55.839 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:29:55.839 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:29:55.839 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:29:55.839 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-05-15T12:29:55.839 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-mon.a.asok log flush 2024-05-15T12:29:55.947 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:29:55.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse4 2024-05-15T12:29:55.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-15T12:29:55.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:29:55.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T12:29:55.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:29:55.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-15T12:29:55.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-15T12:29:55.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:29:55.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:29:55.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:29:55.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-15T12:29:55.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-15T12:29:55.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:29:55.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:29:55.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-15T12:29:55.964 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:29:55.964 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:29:55.964 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:29:55.965 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:29:55.965 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:29:55.965 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:29:55.966 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:29:55.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:29:55.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:29:55.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:29:55.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:29:55.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-15T12:29:55.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-15T12:29:55.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:29:55.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:29:55.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:29:55.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:29:55.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:29:55.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse4 2024-05-15T12:29:55.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-15T12:29:55.967 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:29:55.967 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 63c4d29d-f6f6-43c4-874e-f059b5694c34 2024-05-15T12:29:55.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=63c4d29d-f6f6-43c4-874e-f059b5694c34 2024-05-15T12:29:55.968 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 63c4d29d-f6f6-43c4-874e-f059b5694c34' 2024-05-15T12:29:55.968 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:29:55.988 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDEqkRmBZ8UABAAAUmo/Hhv/NLUKx4F5xfwrg== 2024-05-15T12:29:55.988 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDEqkRmBZ8UABAAAUmo/Hhv/NLUKx4F5xfwrg=="}' 2024-05-15T12:29:55.988 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 63c4d29d-f6f6-43c4-874e-f059b5694c34 -i td/test-erasure-code-plugins/0/new.json 2024-05-15T12:29:56.315 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:29:56.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-15T12:29:56.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 --mkfs --key AQDEqkRmBZ8UABAAAUmo/Hhv/NLUKx4F5xfwrg== --osd-uuid 63c4d29d-f6f6-43c4-874e-f059b5694c34 2024-05-15T12:29:56.363 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:29:56.376+0000 7fc04379b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:29:56.393 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:29:56.404+0000 7fc04379b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:29:56.394 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:29:56.408+0000 7fc04379b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:29:56.394 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:29:56.408+0000 7fc04379b7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:29:56.395 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:29:56.408+0000 7fc04379b7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:29:56.395 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:29:56.408+0000 7fc04379b7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:29:56.432 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:29:56.444+0000 7fc04379b7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-15T12:30:02.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-15T12:30:02.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:30:02.892 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T12:30:02.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T12:30:02.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:30:03.452 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T12:30:03.453 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T12:30:03.453 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:30:03.453 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:30:03.453 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:30:03.453 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 2024-05-15T12:30:03.484 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:30:03.495+0000 7f2a7a3847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:30:03.516 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:30:03.527+0000 7f2a7a3847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:30:03.518 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:30:03.531+0000 7f2a7a3847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:30:03.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T12:30:03.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:30:03.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T12:30:03.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:30:03.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:30:03.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:03.888 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:30:03.889 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:30:03.889 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:03.889 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:04.086 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:30:04.099+0000 7f2a7a3847c0 -1 Falling back to public interface 2024-05-15T12:30:04.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:05.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:05.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:05.319 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:30:05.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:30:05.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:05.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:05.742 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:06.740 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:30:06.751+0000 7f2a7a3847c0 -1 osd.0 0 log_to_monitors true 2024-05-15T12:30:06.743 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:06.743 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:06.744 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:30:06.744 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:30:06.744 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:06.744 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:07.374 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:08.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:08.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:08.376 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:30:08.376 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:30:08.376 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:08.376 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:08.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:09.846 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:09.846 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:09.846 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:30:09.846 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:30:09.847 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:09.847 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:10.322 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:11.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:11.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:11.324 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:30:11.324 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:30:11.324 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:11.324 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:11.798 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:12.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:12.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:12.800 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:30:12.801 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:30:12.801 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:12.801 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:13.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:14.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:14.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:14.236 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:30:14.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:30:14.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:14.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:14.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:15.771 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:15.771 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:15.771 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:30:15.771 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:30:15.771 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:15.772 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:16.205 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:17.207 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:17.207 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:17.207 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:30:17.207 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:30:17.208 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:17.208 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:17.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:18.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:18.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:18.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:30:18.671 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:30:18.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:18.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:19.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:20.101 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:20.101 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:20.101 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:30:20.101 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:30:20.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:20.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:21.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:22.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:22.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:22.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:30:22.337 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:30:22.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:22.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:23.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:24.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:24.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:24.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:30:24.364 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:30:24.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:24.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:25.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:26.247 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:26.248 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:26.248 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:30:26.248 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:30:26.248 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:26.249 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:27.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:28.028 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:28.028 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:28.029 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:30:28.029 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:30:28.029 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:28.029 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:28.949 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:29.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:29.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:29.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:30:29.951 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:30:29.952 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:29.952 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:30.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:31.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:31.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:31.606 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:30:31.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:30:31.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:31.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:32.078 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1695955904,v1:127.0.0.1:6803/1695955904] [v2:127.0.0.1:6804/1695955904,v1:127.0.0.1:6805/1695955904] exists,up 63c4d29d-f6f6-43c4-874e-f059b5694c34 2024-05-15T12:30:32.078 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:30:32.078 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:30:32.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:30:32.083 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-05-15T12:30:32.083 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T12:30:32.083 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T12:30:32.083 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:30:32.083 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:30:32.083 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:30:32.083 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-osd.0.asok 2024-05-15T12:30:32.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-05-15T12:30:32.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-osd.0.asok log flush 2024-05-15T12:30:32.195 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:30:32.208 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2024-05-15T12:30:32.212 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:29:54.460+0000 7fabd4554c80 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-05-15T12:30:32.212 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/osd.0.log 2024-05-15T12:30:32.214 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:30:04.099+0000 7f2a7a3847c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-05-15T12:30:32.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-05-15T12:30:32.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:30:32.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:30:32.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:30:32.215 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:30:32.216 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:30:32.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:30:32.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:30:32.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:30:32.324 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:30:32.325 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:30:32.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:30:32.326 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:30:32.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:30:32.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:30:32.327 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:30:32.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:30:32.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:30:32.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:30:32.329 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:30:32.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:30:32.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:30:32.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:30:32.336 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:30:32.336 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:30:32.336 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:30:32.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:30:32.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:30:32.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:30:32.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-05-15T12:30:32.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-05-15T12:30:32.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-15T12:30:32.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-15T12:30:32.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:30:32.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:30:32.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:30:32.339 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:30:32.339 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:30:32.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:30:32.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:30:32.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:30:32.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:30:32.340 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:30:32.341 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:30:32.341 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:30:32.342 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:30:32.342 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:30:32.343 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:30:32.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:30:32.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:30:32.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:30:32.344 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:30:32.345 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:30:32.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:30:32.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:30:32.347 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:30:32.347 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:30:32.347 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:30:32.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:30:32.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:30:32.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:30:32.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-15T12:30:32.350 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T12:30:32.350 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:30:32.350 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:30:32.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20204 2024-05-15T12:30:32.351 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T12:30:32.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-15T12:30:32.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T12:30:32.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-15T12:30:32.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_generic 2024-05-15T12:30:32.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-15T12:30:32.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T12:30:32.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T12:30:32.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T12:30:32.353 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-15T12:30:32.353 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_generic 2024-05-15T12:30:32.780 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T12:30:32.780 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:30:32.780 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:30:32.781 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:30:32.781 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:30:32.781 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:30:32.781 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:30:32.781 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_generic 2024-05-15T12:30:33.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T12:30:33.009 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T12:30:33.009 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:30:33.009 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:30:33.009 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T12:30:33.010 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T12:30:33.010 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:30:33.010 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:30:33.010 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:30:33.010 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:30:33.010 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:30:33.010 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:30:33.011 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:30:33.011 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:30:33.011 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get fsid 2024-05-15T12:30:33.129 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T12:30:33.129 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:30:33.129 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:30:33.129 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T12:30:33.130 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T12:30:33.130 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:30:33.130 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:30:33.130 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:30:33.130 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:30:33.130 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:30:33.131 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:30:33.131 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:30:33.131 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:30:33.131 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get mon_host 2024-05-15T12:30:33.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-05-15T12:30:33.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-15T12:30:33.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T12:30:33.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T12:30:33.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T12:30:33.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-15T12:30:33.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T12:30:34.156 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T12:30:34.156 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:30:34.156 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:30:34.157 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:30:34.157 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:30:34.157 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:30:34.157 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:30:34.158 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:30:34.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:30:34.222 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-05-15T12:30:34.222 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:30:34.222 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:30:34.223 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:30:34.223 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:30:34.223 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:30:34.223 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:30:34.223 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-05-15T12:30:34.223 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-mon.a.asok log flush 2024-05-15T12:30:34.327 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:30:34.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_generic 2024-05-15T12:30:34.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-15T12:30:34.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:30:34.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T12:30:34.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:30:34.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-15T12:30:34.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-15T12:30:34.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:30:34.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:30:34.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:30:34.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-15T12:30:34.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-15T12:30:34.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:30:34.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:30:34.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-15T12:30:34.344 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:30:34.344 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:30:34.344 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:30:34.345 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:30:34.345 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:30:34.345 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:30:34.346 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:30:34.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:30:34.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:30:34.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:30:34.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:30:34.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-15T12:30:34.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-15T12:30:34.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:30:34.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:30:34.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:30:34.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:30:34.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:30:34.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_generic 2024-05-15T12:30:34.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-15T12:30:34.347 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:30:34.348 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 9c849a00-6a9d-4596-b142-26ac6f695000 2024-05-15T12:30:34.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9c849a00-6a9d-4596-b142-26ac6f695000 2024-05-15T12:30:34.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9c849a00-6a9d-4596-b142-26ac6f695000' 2024-05-15T12:30:34.348 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:30:34.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDqqkRmR+biFhAAZDJ1nfS0h6VbndFvSgzfkg== 2024-05-15T12:30:34.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDqqkRmR+biFhAAZDJ1nfS0h6VbndFvSgzfkg=="}' 2024-05-15T12:30:34.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9c849a00-6a9d-4596-b142-26ac6f695000 -i td/test-erasure-code-plugins/0/new.json 2024-05-15T12:30:34.669 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:30:34.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-15T12:30:34.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic --mkfs --key AQDqqkRmR+biFhAAZDJ1nfS0h6VbndFvSgzfkg== --osd-uuid 9c849a00-6a9d-4596-b142-26ac6f695000 2024-05-15T12:30:34.713 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:30:34.727+0000 7f51f6ac77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:30:34.738 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:30:34.751+0000 7f51f6ac77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:30:34.739 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:30:34.751+0000 7f51f6ac77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:30:34.740 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:30:34.751+0000 7f51f6ac77c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:30:34.740 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:30:34.751+0000 7f51f6ac77c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:30:34.741 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:30:34.751+0000 7f51f6ac77c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:30:34.778 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:30:34.791+0000 7f51f6ac77c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-15T12:30:41.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-15T12:30:41.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:30:41.135 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T12:30:41.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T12:30:41.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:30:41.707 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T12:30:41.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T12:30:41.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic 2024-05-15T12:30:41.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:30:41.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:30:41.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:30:41.738 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:30:41.751+0000 7f8c733a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:30:41.741 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:30:41.755+0000 7f8c733a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:30:41.742 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:30:41.755+0000 7f8c733a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:30:42.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T12:30:42.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:30:42.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T12:30:42.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:30:42.150 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:30:42.151 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:30:42.151 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:42.151 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:30:42.151 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:42.151 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:42.302 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:30:42.315+0000 7f8c733a97c0 -1 Falling back to public interface 2024-05-15T12:30:42.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:43.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:43.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:43.641 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:30:43.642 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:30:43.642 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:43.642 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:44.096 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:44.925 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:30:44.939+0000 7f8c733a97c0 -1 osd.0 0 log_to_monitors true 2024-05-15T12:30:45.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:45.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:45.098 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:30:45.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:30:45.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:45.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:45.650 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:46.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:46.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:46.652 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:30:46.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:30:46.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:46.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:47.105 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:48.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:48.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:48.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:30:48.108 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:30:48.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:48.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:48.563 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:49.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:49.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:49.565 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:30:49.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:30:49.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:49.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:50.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:51.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:51.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:51.128 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:30:51.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:30:51.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:51.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:51.552 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:52.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:52.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:52.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:30:52.555 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:30:52.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:52.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:53.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:54.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:54.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:54.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:30:54.086 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:30:54.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:54.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:54.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:55.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:55.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:55.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:30:55.522 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:30:55.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:55.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:56.096 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:57.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:57.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:57.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:30:57.099 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:30:57.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:57.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:57.547 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:30:58.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:30:58.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:30:58.549 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:30:58.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:30:58.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:30:58.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:30:59.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:00.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:00.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:00.487 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:31:00.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:31:00.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:00.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:01.759 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:02.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:02.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:02.761 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:31:02.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:31:02.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:02.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:03.688 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:04.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:04.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:04.690 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:31:04.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:31:04.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:04.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:05.498 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:06.500 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:06.500 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:06.501 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:31:06.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:31:06.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:06.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:07.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:08.345 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:08.345 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:08.346 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:31:08.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:31:08.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:08.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:09.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:10.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:10.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:10.309 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:31:10.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:31:10.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:10.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:10.742 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3585193307,v1:127.0.0.1:6803/3585193307] [v2:127.0.0.1:6804/3585193307,v1:127.0.0.1:6805/3585193307] exists,up 9c849a00-6a9d-4596-b142-26ac6f695000 2024-05-15T12:31:10.743 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:31:10.743 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:31:10.743 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:31:10.743 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-05-15T12:31:10.743 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T12:31:10.744 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T12:31:10.744 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:31:10.744 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:31:10.744 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:31:10.745 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-osd.0.asok 2024-05-15T12:31:10.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-05-15T12:31:10.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-osd.0.asok log flush 2024-05-15T12:31:10.857 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:31:10.870 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2024-05-15T12:31:10.875 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:30:32.835+0000 7f2dfedffc80 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-05-15T12:31:10.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/osd.0.log 2024-05-15T12:31:10.876 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:30:42.315+0000 7f8c733a97c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-05-15T12:31:10.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-05-15T12:31:10.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:31:10.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:31:10.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:31:10.878 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:31:10.878 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:31:10.878 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:31:10.878 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:31:10.878 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:31:10.987 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:31:10.987 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:31:10.988 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:31:10.989 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:31:10.990 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:31:10.990 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:31:10.990 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:31:10.991 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:31:10.991 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:31:10.991 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:31:10.991 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:31:10.992 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:31:10.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:31:10.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:31:10.999 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:31:10.999 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:31:10.999 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:31:10.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:31:11.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:31:11.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:31:11.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-05-15T12:31:11.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-05-15T12:31:11.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-15T12:31:11.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-15T12:31:11.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:31:11.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:31:11.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:31:11.002 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:31:11.002 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:31:11.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:31:11.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:31:11.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:31:11.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:31:11.003 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:31:11.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:31:11.005 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:31:11.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:31:11.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:31:11.006 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:31:11.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:31:11.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:31:11.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:31:11.007 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:31:11.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:31:11.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:31:11.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:31:11.010 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:31:11.010 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:31:11.011 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:31:11.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:31:11.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:31:11.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:31:11.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-15T12:31:11.013 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T12:31:11.013 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:31:11.013 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:31:11.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20204 2024-05-15T12:31:11.015 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T12:31:11.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-15T12:31:11.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T12:31:11.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-15T12:31:11.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse3 2024-05-15T12:31:11.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-15T12:31:11.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T12:31:11.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T12:31:11.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T12:31:11.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-15T12:31:11.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse3 2024-05-15T12:31:11.422 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T12:31:11.423 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:31:11.423 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:31:11.423 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:31:11.423 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:31:11.424 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:31:11.424 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:31:11.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse3 2024-05-15T12:31:11.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T12:31:11.687 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T12:31:11.687 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:31:11.687 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:31:11.687 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T12:31:11.688 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T12:31:11.688 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:31:11.688 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:31:11.688 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:31:11.689 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:31:11.689 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:31:11.689 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:31:11.689 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:31:11.689 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:31:11.690 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get fsid 2024-05-15T12:31:11.807 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T12:31:11.807 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:31:11.807 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:31:11.807 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T12:31:11.808 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T12:31:11.808 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:31:11.808 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:31:11.808 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:31:11.808 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:31:11.808 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:31:11.809 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:31:11.809 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:31:11.809 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:31:11.809 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get mon_host 2024-05-15T12:31:11.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-05-15T12:31:11.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-15T12:31:11.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T12:31:11.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T12:31:11.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T12:31:11.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-15T12:31:11.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T12:31:12.856 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T12:31:12.857 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:31:12.857 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:31:12.857 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:31:12.857 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:31:12.857 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:31:12.858 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:31:12.858 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:31:12.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:31:12.925 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-05-15T12:31:12.925 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:31:12.925 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:31:12.925 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:31:12.925 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:31:12.925 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:31:12.926 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:31:12.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-05-15T12:31:12.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-mon.a.asok log flush 2024-05-15T12:31:13.027 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:31:13.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse3 2024-05-15T12:31:13.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-15T12:31:13.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:31:13.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T12:31:13.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:31:13.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-15T12:31:13.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-15T12:31:13.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:31:13.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:31:13.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:31:13.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-15T12:31:13.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-15T12:31:13.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:31:13.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:31:13.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-15T12:31:13.044 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:31:13.044 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:31:13.044 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:31:13.044 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:31:13.044 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:31:13.044 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:31:13.045 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:31:13.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:31:13.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:31:13.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:31:13.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:31:13.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-15T12:31:13.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-15T12:31:13.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:31:13.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:31:13.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:31:13.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:31:13.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:31:13.046 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse3 2024-05-15T12:31:13.046 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-15T12:31:13.046 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:31:13.048 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f5394b7e-367e-4375-b3c0-7146196f1cde 2024-05-15T12:31:13.048 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f5394b7e-367e-4375-b3c0-7146196f1cde' 2024-05-15T12:31:13.048 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 f5394b7e-367e-4375-b3c0-7146196f1cde 2024-05-15T12:31:13.048 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:31:13.070 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQARq0Rmen8KBRAA82eJTzZbUH7EKsrDuPMZvA== 2024-05-15T12:31:13.070 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQARq0Rmen8KBRAA82eJTzZbUH7EKsrDuPMZvA=="}' 2024-05-15T12:31:13.070 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f5394b7e-367e-4375-b3c0-7146196f1cde -i td/test-erasure-code-plugins/0/new.json 2024-05-15T12:31:13.380 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:31:13.395 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-15T12:31:13.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 --mkfs --key AQARq0Rmen8KBRAA82eJTzZbUH7EKsrDuPMZvA== --osd-uuid f5394b7e-367e-4375-b3c0-7146196f1cde 2024-05-15T12:31:13.424 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:13.434+0000 7f5b6edfa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:31:13.458 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:13.470+0000 7f5b6edfa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:31:13.459 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:13.470+0000 7f5b6edfa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:31:13.460 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:13.470+0000 7f5b6edfa7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:31:13.460 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:13.470+0000 7f5b6edfa7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:31:13.460 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:13.470+0000 7f5b6edfa7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:31:13.497 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:13.510+0000 7f5b6edfa7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-15T12:31:19.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-15T12:31:19.646 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:31:19.647 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T12:31:19.648 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T12:31:19.648 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:31:20.208 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T12:31:20.208 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T12:31:20.208 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:31:20.208 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 2024-05-15T12:31:20.209 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:31:20.209 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:31:20.239 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:20.250+0000 7fe8130b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:31:20.253 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:20.266+0000 7fe8130b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:31:20.254 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:20.266+0000 7fe8130b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:31:20.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T12:31:20.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:31:20.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T12:31:20.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:31:20.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:31:20.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:20.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:31:20.618 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:31:20.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:20.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:20.818 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:20.830+0000 7fe8130b47c0 -1 Falling back to public interface 2024-05-15T12:31:21.025 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:22.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:22.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:22.027 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:31:22.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:31:22.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:22.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:22.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:23.360 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:23.370+0000 7fe8130b47c0 -1 osd.0 0 log_to_monitors true 2024-05-15T12:31:23.456 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:23.456 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:23.456 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:31:23.457 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:31:23.457 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:23.457 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:23.983 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:24.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:24.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:24.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:31:24.985 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:31:24.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:24.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:25.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:26.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:26.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:26.427 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:31:26.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:31:26.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:26.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:26.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:27.857 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:27.857 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:27.857 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:31:27.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:31:27.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:27.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:28.278 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:29.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:29.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:29.280 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:31:29.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:31:29.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:29.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:29.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:30.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:30.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:30.703 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:31:30.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:31:30.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:30.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:31.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:32.180 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:32.180 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:32.181 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:31:32.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:31:32.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:32.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:32.617 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:33.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:33.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:33.620 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:31:33.620 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:31:33.620 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:33.620 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:34.146 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:35.148 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:35.148 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:35.148 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:31:35.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:31:35.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:35.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:35.688 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:36.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:36.690 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:31:36.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:36.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:31:36.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:36.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:37.277 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:38.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:38.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:38.279 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:31:38.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:31:38.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:38.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:39.187 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:40.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:40.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:40.189 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:31:40.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:31:40.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:40.190 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:40.992 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:41.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:41.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:41.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:31:41.994 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:31:41.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:41.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:42.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:43.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:43.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:43.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:31:43.815 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:31:43.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:43.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:44.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:45.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:45.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:45.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:31:45.319 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:31:45.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:45.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:46.127 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:47.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:47.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:47.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:31:47.130 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:31:47.130 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:47.130 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:47.583 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1697856525,v1:127.0.0.1:6803/1697856525] [v2:127.0.0.1:6804/1697856525,v1:127.0.0.1:6805/1697856525] exists,up f5394b7e-367e-4375-b3c0-7146196f1cde 2024-05-15T12:31:47.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:31:47.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:31:47.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:31:47.584 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-05-15T12:31:47.585 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T12:31:47.585 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T12:31:47.586 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:31:47.586 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:31:47.586 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:31:47.586 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-osd.0.asok 2024-05-15T12:31:47.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-05-15T12:31:47.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-osd.0.asok log flush 2024-05-15T12:31:47.688 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:31:47.699 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2024-05-15T12:31:47.704 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:31:11.474+0000 7f89de0c3c80 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-05-15T12:31:47.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/osd.0.log 2024-05-15T12:31:47.706 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:31:20.830+0000 7fe8130b47c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-05-15T12:31:47.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-05-15T12:31:47.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:31:47.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:31:47.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:31:47.708 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:31:47.708 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:31:47.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:31:47.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:31:47.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:31:47.817 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:31:47.817 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:31:47.818 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:31:47.818 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:31:47.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:31:47.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:31:47.820 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:31:47.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:31:47.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:31:47.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:31:47.821 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:31:47.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:31:47.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:31:47.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:31:47.828 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:31:47.828 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:31:47.829 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:31:47.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:31:47.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:31:47.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:31:47.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-05-15T12:31:47.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-05-15T12:31:47.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-15T12:31:47.831 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-15T12:31:47.831 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:31:47.831 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:31:47.831 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:31:47.831 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:31:47.831 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:31:47.831 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:31:47.831 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:31:47.831 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:31:47.832 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:31:47.832 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:31:47.833 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:31:47.834 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:31:47.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:31:47.835 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:31:47.835 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:31:47.836 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:31:47.836 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:31:47.836 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:31:47.836 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:31:47.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:31:47.838 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:31:47.838 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:31:47.839 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:31:47.840 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:31:47.840 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:31:47.840 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:31:47.841 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:31:47.841 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:31:47.841 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-15T12:31:47.842 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T12:31:47.842 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:31:47.842 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:31:47.842 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20204 2024-05-15T12:31:47.844 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T12:31:47.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-15T12:31:47.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T12:31:47.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-15T12:31:47.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse4 2024-05-15T12:31:47.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-15T12:31:47.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T12:31:47.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T12:31:47.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T12:31:47.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-15T12:31:47.845 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse4 2024-05-15T12:31:48.241 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T12:31:48.241 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:31:48.241 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:31:48.241 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:31:48.242 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:31:48.242 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:31:48.242 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:31:48.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse4 2024-05-15T12:31:48.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T12:31:48.428 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T12:31:48.428 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:31:48.428 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:31:48.428 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T12:31:48.428 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T12:31:48.429 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:31:48.429 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:31:48.429 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:31:48.429 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:31:48.429 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:31:48.429 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:31:48.430 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:31:48.430 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:31:48.430 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get fsid 2024-05-15T12:31:48.548 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T12:31:48.548 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:31:48.548 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:31:48.548 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T12:31:48.548 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T12:31:48.549 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:31:48.549 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:31:48.549 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:31:48.549 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:31:48.549 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:31:48.550 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:31:48.550 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:31:48.550 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:31:48.550 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20204/ceph-mon.a.asok config get mon_host 2024-05-15T12:31:48.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-05-15T12:31:48.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-15T12:31:48.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T12:31:48.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T12:31:48.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T12:31:48.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-15T12:31:48.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T12:31:49.560 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T12:31:49.561 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:31:49.561 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:31:49.561 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:31:49.561 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:31:49.561 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:31:49.561 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:31:49.562 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:31:49.563 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:31:49.634 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-05-15T12:31:49.634 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:31:49.634 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:31:49.635 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:31:49.635 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:31:49.635 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:31:49.635 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-mon.a.asok 2024-05-15T12:31:49.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-05-15T12:31:49.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-mon.a.asok log flush 2024-05-15T12:31:49.742 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:31:49.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse4 2024-05-15T12:31:49.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-15T12:31:49.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:31:49.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T12:31:49.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:31:49.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-15T12:31:49.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-15T12:31:49.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:31:49.759 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:31:49.759 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:31:49.759 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-15T12:31:49.759 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-15T12:31:49.759 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:31:49.759 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:31:49.759 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-15T12:31:49.759 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:31:49.759 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:31:49.759 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:31:49.761 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:31:49.761 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:31:49.761 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:31:49.761 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:31:49.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' 2024-05-15T12:31:49.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:31:49.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:31:49.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:31:49.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-15T12:31:49.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-15T12:31:49.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:31:49.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:31:49.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:31:49.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:31:49.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:31:49.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse4 2024-05-15T12:31:49.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-15T12:31:49.762 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:31:49.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ea2047d9-c85a-4d5b-9f44-855e5bfaa98c 2024-05-15T12:31:49.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ea2047d9-c85a-4d5b-9f44-855e5bfaa98c' 2024-05-15T12:31:49.763 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 ea2047d9-c85a-4d5b-9f44-855e5bfaa98c 2024-05-15T12:31:49.763 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:31:49.784 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA1q0RmM9mZLxAAWU94LaBEoPl9BAxkxUBVuA== 2024-05-15T12:31:49.784 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA1q0RmM9mZLxAAWU94LaBEoPl9BAxkxUBVuA=="}' 2024-05-15T12:31:49.784 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ea2047d9-c85a-4d5b-9f44-855e5bfaa98c -i td/test-erasure-code-plugins/0/new.json 2024-05-15T12:31:50.133 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:31:50.148 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-15T12:31:50.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 --mkfs --key AQA1q0RmM9mZLxAAWU94LaBEoPl9BAxkxUBVuA== --osd-uuid ea2047d9-c85a-4d5b-9f44-855e5bfaa98c 2024-05-15T12:31:50.188 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:50.198+0000 7f68961e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:31:50.199 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:50.210+0000 7f68961e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:31:50.201 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:50.214+0000 7f68961e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:31:50.201 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:50.214+0000 7f68961e97c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:31:50.201 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:50.214+0000 7f68961e97c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:31:50.201 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:50.214+0000 7f68961e97c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-15T12:31:50.239 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:50.250+0000 7f68961e97c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-15T12:31:56.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-15T12:31:56.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:31:56.503 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T12:31:56.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T12:31:56.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:31:57.048 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T12:31:57.049 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T12:31:57.049 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=632de428-72db-451e-809b-106dd1d365f0 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20204/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 2024-05-15T12:31:57.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:31:57.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:31:57.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:31:57.073 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:57.086+0000 7ff19c5247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:31:57.105 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:57.118+0000 7ff19c5247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:31:57.106 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:57.118+0000 7ff19c5247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:31:57.460 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T12:31:57.460 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:31:57.460 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T12:31:57.460 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:31:57.460 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:31:57.460 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:57.460 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:31:57.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:31:57.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:57.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:57.650 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:31:57.662+0000 7ff19c5247c0 -1 Falling back to public interface 2024-05-15T12:31:57.870 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:31:58.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:31:58.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:31:58.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:31:58.873 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:31:58.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:31:58.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:31:59.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:00.195 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:32:00.206+0000 7ff19c5247c0 -1 osd.0 0 log_to_monitors true 2024-05-15T12:32:00.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:00.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:00.287 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:32:00.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:32:00.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:00.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:00.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:01.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:01.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:01.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:32:01.860 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:32:01.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:01.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:02.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:03.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:03.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:03.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:32:03.299 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:32:03.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:03.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:03.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:04.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:04.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:04.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:32:04.732 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:32:04.732 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:04.732 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:05.303 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:06.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:06.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:06.305 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:32:06.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:32:06.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:06.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:06.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:07.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:07.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:07.768 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:32:07.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:32:07.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:07.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:08.374 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:09.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:09.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:09.376 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:32:09.376 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:32:09.376 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:09.376 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:09.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:10.799 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:10.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:10.800 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:32:10.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:32:10.801 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:10.801 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:11.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:12.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:12.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:12.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:32:12.338 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:32:12.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:12.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:12.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:13.771 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:13.771 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:13.771 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:32:13.772 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:32:13.772 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:13.772 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:14.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:15.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:15.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:15.593 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:32:15.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:32:15.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:15.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:16.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:17.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:17.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:17.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:32:17.369 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:32:17.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:17.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:18.145 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:19.147 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:19.147 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:19.147 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:32:19.147 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:32:19.148 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:19.148 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:20.278 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:21.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:21.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:21.280 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:32:21.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:32:21.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:21.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:22.122 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:23.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:23.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:23.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:32:23.125 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:32:23.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:23.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:23.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:24.996 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:24.996 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:24.996 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:32:24.996 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:32:24.996 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:24.997 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:25.461 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1105883831,v1:127.0.0.1:6803/1105883831] [v2:127.0.0.1:6804/1105883831,v1:127.0.0.1:6805/1105883831] exists,up ea2047d9-c85a-4d5b-9f44-855e5bfaa98c 2024-05-15T12:32:25.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:32:25.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:32:25.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:32:25.463 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-05-15T12:32:25.463 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T12:32:25.463 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T12:32:25.464 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:32:25.464 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:32:25.464 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:32:25.464 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20204/ceph-osd.0.asok 2024-05-15T12:32:25.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-05-15T12:32:25.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20204/ceph-osd.0.asok log flush 2024-05-15T12:32:25.574 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:32:25.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2024-05-15T12:32:25.590 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:31:48.294+0000 7ff856d35c80 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-05-15T12:32:25.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/osd.0.log 2024-05-15T12:32:25.592 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:31:57.662+0000 7ff19c5247c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-05-15T12:32:25.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-05-15T12:32:25.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:32:25.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:32:25.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:32:25.593 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:32:25.593 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:32:25.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:32:25.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:32:25.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:32:25.903 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:32:25.903 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:32:25.904 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:32:25.905 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:32:25.906 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:32:25.906 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:32:25.906 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:32:25.907 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:32:25.907 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:32:25.908 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:32:25.908 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:32:25.908 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:32:25.910 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:32:25.910 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:32:25.917 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:32:25.918 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:32:25.918 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:32:25.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:32:25.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:32:25.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:32:25.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:54: TEST_preload_warning: return 0 2024-05-15T12:32:25.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-05-15T12:32:25.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/test-erasure-code-plugins 0 2024-05-15T12:32:25.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-15T12:32:25.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-05-15T12:32:25.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-15T12:32:25.920 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:32:25.920 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:32:25.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:32:25.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:32:25.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:32:25.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:32:25.923 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:32:25.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:32:25.924 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:32:25.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:32:25.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:32:25.925 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:32:25.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:32:25.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:32:25.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:32:25.927 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:32:25.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:32:25.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-05-15T12:32:25.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-15T12:32:25.931 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:32:25.932 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:32:25.932 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20204 2024-05-15T12:32:25.932 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20204 2024-05-15T12:32:25.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:32:25.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:32:25.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-05-15T12:32:25.934 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-15T12:32:25.934 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-15T12:32:25.946 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code.sh... 2024-05-15T12:32:25.946 DEBUG:teuthology.orchestra.run.smithi005:workunit test erasure-code/test-erasure-code.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae2481253f9b2fa3451954e615eedb67ebdd5314 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh 2024-05-15T12:32:26.000 INFO:tasks.workunit.client.0.smithi005.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-05-15T12:32:26.006 INFO:tasks.workunit.client.0.smithi005.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-05-15T12:32:26.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-05-15T12:32:26.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-05-15T12:32:26.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-05-15T12:32:26.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-05-15T12:32:26.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-05-15T12:32:26.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-05-15T12:32:26.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-05-15T12:32:26.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-05-15T12:32:26.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/test-erasure-code 2024-05-15T12:32:26.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:22: run: local dir=td/test-erasure-code 2024-05-15T12:32:26.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:23: run: shift 2024-05-15T12:32:26.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: export CEPH_MON=127.0.0.1:7101 2024-05-15T12:32:26.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: CEPH_MON=127.0.0.1:7101 2024-05-15T12:32:26.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:26: run: export CEPH_ARGS 2024-05-15T12:32:26.007 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: uuidgen 2024-05-15T12:32:26.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: CEPH_ARGS+='--fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none ' 2024-05-15T12:32:26.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-15T12:32:26.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:30: run: setup td/test-erasure-code 2024-05-15T12:32:26.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code 2024-05-15T12:32:26.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code 2024-05-15T12:32:26.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-05-15T12:32:26.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:32:26.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-05-15T12:32:26.009 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:32:26.009 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:32:26.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:32:26.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:32:26.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:32:26.010 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:32:26.011 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:32:26.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:32:26.012 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:32:26.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:32:26.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:32:26.013 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:32:26.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:32:26.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:32:26.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:32:26.015 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:32:26.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:32:26.017 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:32:26.017 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-05-15T12:32:26.018 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:32:26.018 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:32:26.018 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:32:26.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48542 2024-05-15T12:32:26.019 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:32:26.019 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:32:26.019 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code 2024-05-15T12:32:26.021 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T12:32:26.021 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:32:26.021 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:32:26.021 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.48542 2024-05-15T12:32:26.023 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T12:32:26.023 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-05-15T12:32:26.023 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-05-15T12:32:26.023 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T12:32:26.023 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code 1' TERM HUP INT 2024-05-15T12:32:26.023 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:31: run: run_mon td/test-erasure-code a 2024-05-15T12:32:26.023 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code 2024-05-15T12:32:26.024 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T12:32:26.024 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T12:32:26.024 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T12:32:26.024 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code/a 2024-05-15T12:32:26.024 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code/a --run-dir=td/test-erasure-code 2024-05-15T12:32:26.427 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T12:32:26.427 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:32:26.427 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:32:26.428 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:32:26.428 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:32:26.428 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:32:26.428 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:32:26.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code/a '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code/log --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-15T12:32:26.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T12:32:26.640 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T12:32:26.640 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:32:26.640 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:32:26.640 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T12:32:26.641 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T12:32:26.641 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:32:26.641 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:32:26.641 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:32:26.641 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:32:26.641 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:32:26.641 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:32:26.641 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48542/ceph-mon.a.asok 2024-05-15T12:32:26.642 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:32:26.642 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48542/ceph-mon.a.asok config get fsid 2024-05-15T12:32:26.759 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T12:32:26.759 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:32:26.759 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:32:26.759 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T12:32:26.760 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:32:26.760 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:32:26.760 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:32:26.760 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T12:32:26.761 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:32:26.761 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:32:26.761 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:32:26.761 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48542/ceph-mon.a.asok 2024-05-15T12:32:26.761 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:32:26.761 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48542/ceph-mon.a.asok config get mon_host 2024-05-15T12:32:26.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:32: run: run_mgr td/test-erasure-code x 2024-05-15T12:32:26.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code 2024-05-15T12:32:26.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T12:32:26.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T12:32:26.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T12:32:26.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code/x 2024-05-15T12:32:26.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T12:32:27.776 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T12:32:27.776 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:32:27.776 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:32:27.776 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:32:27.776 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:32:27.776 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:32:27.777 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:32:27.777 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:32:27.777 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code/x '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:32:27.812 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: get_asok_path mon.a 2024-05-15T12:32:27.812 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:32:27.812 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:32:27.812 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:32:27.812 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:32:27.813 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:32:27.813 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48542/ceph-mon.a.asok 2024-05-15T12:32:27.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: CEPH_ARGS= 2024-05-15T12:32:27.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: ceph --admin-daemon /tmp/ceph-asok.48542/ceph-mon.a.asok log flush 2024-05-15T12:32:27.913 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:32:27.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:35: run: grep 'load: jerasure.*lrc' td/test-erasure-code/mon.a.log 2024-05-15T12:32:27.929 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:32:26.481+0000 7f484986bc80 0 load: jerasure load: lrc 2024-05-15T12:32:27.930 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: seq 0 10 2024-05-15T12:32:27.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-15T12:32:27.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 0 2024-05-15T12:32:27.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-15T12:32:27.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:32:27.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T12:32:27.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:32:27.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/0 2024-05-15T12:32:27.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-15T12:32:27.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:32:27.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:32:27.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:32:27.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/0' 2024-05-15T12:32:27.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/0/journal' 2024-05-15T12:32:27.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:32:27.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:32:27.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-15T12:32:27.934 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:32:27.934 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:32:27.934 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:32:27.934 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:32:27.934 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:32:27.934 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:32:27.938 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:32:27.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:32:27.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:32:27.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:32:27.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:32:27.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-15T12:32:27.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-15T12:32:27.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:32:27.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:32:27.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:32:27.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:32:27.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:32:27.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:32:27.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/0 2024-05-15T12:32:27.939 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:32:27.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=42bf5fe3-6a32-456e-91b6-7caa6d573fce 2024-05-15T12:32:27.939 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 42bf5fe3-6a32-456e-91b6-7caa6d573fce 2024-05-15T12:32:27.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 42bf5fe3-6a32-456e-91b6-7caa6d573fce' 2024-05-15T12:32:27.940 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:32:27.953 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBbq0RmJoW7ORAAcfPj7GZ88Gw7Qk+Joe+uqw== 2024-05-15T12:32:27.954 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBbq0RmJoW7ORAAcfPj7GZ88Gw7Qk+Joe+uqw=="}' 2024-05-15T12:32:27.954 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 42bf5fe3-6a32-456e-91b6-7caa6d573fce -i td/test-erasure-code/0/new.json 2024-05-15T12:32:28.284 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:32:28.303 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/0/new.json 2024-05-15T12:32:28.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBbq0RmJoW7ORAAcfPj7GZ88Gw7Qk+Joe+uqw== --osd-uuid 42bf5fe3-6a32-456e-91b6-7caa6d573fce 2024-05-15T12:32:28.337 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:32:28.349+0000 7fee1e7647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:32:28.361 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:32:28.373+0000 7fee1e7647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:32:28.362 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:32:28.373+0000 7fee1e7647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:32:28.362 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:32:28.373+0000 7fee1e7647c0 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-05-15T12:32:28.363 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:32:28.373+0000 7fee1e7647c0 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-05-15T12:32:28.363 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:32:28.373+0000 7fee1e7647c0 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-05-15T12:32:28.400 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:32:28.413+0000 7fee1e7647c0 -1 bluestore(td/test-erasure-code/0) _read_fsid unparsable uuid 2024-05-15T12:32:34.581 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/0/keyring 2024-05-15T12:32:34.581 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:32:34.583 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T12:32:34.583 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T12:32:34.583 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:32:35.128 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T12:32:35.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T12:32:35.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:32:35.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:32:35.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:32:35.131 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:32:35.160 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:32:35.173+0000 7f4f1b28d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:32:35.191 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:32:35.201+0000 7f4f1b28d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:32:35.192 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:32:35.205+0000 7f4f1b28d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:32:35.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T12:32:35.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:32:35.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T12:32:35.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:32:35.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:32:35.565 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:32:35.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:35.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:32:35.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:35.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:35.741 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:32:35.753+0000 7f4f1b28d7c0 -1 Falling back to public interface 2024-05-15T12:32:35.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:36.983 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:36.983 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:36.984 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:32:36.984 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:32:36.984 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:36.984 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:37.406 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:38.369 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:32:38.381+0000 7f4f1b28d7c0 -1 osd.0 0 log_to_monitors true 2024-05-15T12:32:38.407 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:38.407 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:38.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:32:38.408 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:32:38.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:38.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:39.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:40.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:40.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:40.014 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:32:40.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:32:40.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:40.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:40.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:41.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:41.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:41.571 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:32:41.572 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:32:41.572 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:41.572 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:41.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:43.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:43.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:43.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:32:43.000 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:32:43.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:43.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:43.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:44.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:44.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:44.492 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:32:44.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:32:44.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:44.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:44.952 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:45.954 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:45.954 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:45.954 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:32:45.955 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:32:45.955 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:45.955 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:46.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:47.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:47.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:47.486 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:32:47.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:32:47.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:47.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:47.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:48.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:48.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:48.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:32:48.960 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:32:48.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:48.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:49.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:50.539 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:50.539 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:50.539 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:32:50.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:32:50.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:50.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:51.038 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:52.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:52.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:52.040 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:32:52.041 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:32:52.041 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:52.041 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:52.653 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:53.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:53.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:53.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:32:53.656 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:32:53.656 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:53.656 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:54.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:55.284 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:55.284 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:55.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:32:55.285 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:32:55.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:55.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:56.121 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:57.122 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:57.123 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:57.123 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:32:57.123 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:32:57.123 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:57.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:32:58.324 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:32:59.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:32:59.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:32:59.326 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:32:59.326 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:32:59.326 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:32:59.326 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:33:00.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:01.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:01.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:01.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:33:01.246 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:33:01.247 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:01.247 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:33:02.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:03.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:03.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:03.056 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:33:03.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:33:03.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:03.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:33:03.482 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/229554800,v1:127.0.0.1:6803/229554800] [v2:127.0.0.1:6804/229554800,v1:127.0.0.1:6805/229554800] exists,up 42bf5fe3-6a32-456e-91b6-7caa6d573fce 2024-05-15T12:33:03.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:33:03.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:33:03.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:33:03.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-15T12:33:03.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 1 2024-05-15T12:33:03.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-15T12:33:03.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:33:03.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-15T12:33:03.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:33:03.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/1 2024-05-15T12:33:03.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-15T12:33:03.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:33:03.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:33:03.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:33:03.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/1' 2024-05-15T12:33:03.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/1/journal' 2024-05-15T12:33:03.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:33:03.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:33:03.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-15T12:33:03.486 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:33:03.486 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:33:03.486 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:33:03.486 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:33:03.486 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:33:03.486 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:33:03.486 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:33:03.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:33:03.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:33:03.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:33:03.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:33:03.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-15T12:33:03.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-15T12:33:03.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:33:03.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:33:03.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:33:03.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:33:03.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:33:03.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:33:03.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/1 2024-05-15T12:33:03.487 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:33:03.488 INFO:tasks.workunit.client.0.smithi005.stdout:add osd1 16626615-be19-451b-8b62-a6c9ddc65676 2024-05-15T12:33:03.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=16626615-be19-451b-8b62-a6c9ddc65676 2024-05-15T12:33:03.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 16626615-be19-451b-8b62-a6c9ddc65676' 2024-05-15T12:33:03.488 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:33:03.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB/q0Rmz9krHxAAJeo4m+9AnWCftJk8qvcCfg== 2024-05-15T12:33:03.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB/q0Rmz9krHxAAJeo4m+9AnWCftJk8qvcCfg=="}' 2024-05-15T12:33:03.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 16626615-be19-451b-8b62-a6c9ddc65676 -i td/test-erasure-code/1/new.json 2024-05-15T12:33:04.073 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:33:04.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/1/new.json 2024-05-15T12:33:04.091 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB/q0Rmz9krHxAAJeo4m+9AnWCftJk8qvcCfg== --osd-uuid 16626615-be19-451b-8b62-a6c9ddc65676 2024-05-15T12:33:04.128 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:04.141+0000 7f33373f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:33:04.130 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:04.141+0000 7f33373f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:33:04.131 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:04.145+0000 7f33373f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:33:04.132 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:04.145+0000 7f33373f57c0 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-05-15T12:33:04.132 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:04.145+0000 7f33373f57c0 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-05-15T12:33:04.132 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:04.145+0000 7f33373f57c0 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-05-15T12:33:04.172 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:04.185+0000 7f33373f57c0 -1 bluestore(td/test-erasure-code/1) _read_fsid unparsable uuid 2024-05-15T12:33:10.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/1/keyring 2024-05-15T12:33:10.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:33:10.826 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd1 key to auth repository 2024-05-15T12:33:10.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-15T12:33:10.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:33:11.395 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.1 2024-05-15T12:33:11.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-15T12:33:11.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:33:11.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:33:11.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:33:11.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:33:11.422 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:11.433+0000 7fda958427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:33:11.424 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:11.437+0000 7fda958427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:33:11.425 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:11.437+0000 7fda958427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:33:11.848 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-15T12:33:11.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:33:11.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-15T12:33:11.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:33:11.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:33:11.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:11.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:33:11.849 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:33:11.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:11.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:33:11.981 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:11.993+0000 7fda958427c0 -1 Falling back to public interface 2024-05-15T12:33:12.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:13.300 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:13.301 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:13.301 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:33:13.301 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:33:13.302 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:13.302 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:33:13.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:14.548 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:14.561+0000 7fda958427c0 -1 osd.1 0 log_to_monitors true 2024-05-15T12:33:14.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:14.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:14.764 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:33:14.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:33:14.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:14.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:33:15.277 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:16.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:16.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:16.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:33:16.279 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:33:16.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:16.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:33:16.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:17.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:17.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:17.703 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:33:17.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:33:17.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:17.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:33:18.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:19.146 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:19.146 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:19.146 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:33:19.146 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:33:19.147 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:19.147 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:33:19.683 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:20.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:20.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:20.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:33:20.686 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:33:20.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:20.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:33:21.216 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:22.218 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:22.218 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:22.219 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:33:22.219 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:33:22.219 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:22.219 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:33:22.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:23.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:23.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:23.729 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:33:23.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:33:23.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:23.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:33:24.216 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:25.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:25.218 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:25.218 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:33:25.218 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:33:25.218 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:25.219 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:33:25.725 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:26.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:26.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:26.727 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:33:26.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:33:26.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:26.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:33:27.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:28.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:28.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:28.186 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:33:28.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:33:28.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:28.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:33:28.759 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:29.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:29.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:29.762 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:33:29.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:33:29.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:29.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:33:30.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:31.559 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:31.559 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:31.559 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:33:31.560 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:33:31.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:31.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:33:32.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:33.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:33.271 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:33:33.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:33.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:33:33.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:33.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:33:34.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:35.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:35.431 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:33:35.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:35.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:33:35.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:35.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:33:36.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:37.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:37.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:37.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:33:37.449 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:33:37.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:37.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:33:38.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:39.467 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:39.467 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:39.467 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:33:39.468 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:33:39.468 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:39.468 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:33:39.925 INFO:tasks.workunit.client.0.smithi005.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3859204504,v1:127.0.0.1:6811/3859204504] [v2:127.0.0.1:6812/3859204504,v1:127.0.0.1:6813/3859204504] exists,up 16626615-be19-451b-8b62-a6c9ddc65676 2024-05-15T12:33:39.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:33:39.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:33:39.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:33:39.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-15T12:33:39.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 2 2024-05-15T12:33:39.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-15T12:33:39.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:33:39.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-15T12:33:39.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:33:39.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/2 2024-05-15T12:33:39.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-15T12:33:39.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:33:39.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:33:39.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:33:39.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/2' 2024-05-15T12:33:39.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/2/journal' 2024-05-15T12:33:39.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:33:39.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:33:39.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-15T12:33:39.929 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:33:39.929 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:33:39.929 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:33:39.930 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:33:39.930 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:33:39.931 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:33:39.931 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:33:39.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:33:39.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:33:39.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:33:39.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:33:39.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-15T12:33:39.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-15T12:33:39.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:33:39.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:33:39.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:33:39.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:33:39.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:33:39.932 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:33:39.932 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/2 2024-05-15T12:33:39.932 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:33:39.932 INFO:tasks.workunit.client.0.smithi005.stdout:add osd2 73454772-dc58-46ec-9840-9ab2a8682b05 2024-05-15T12:33:39.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=73454772-dc58-46ec-9840-9ab2a8682b05 2024-05-15T12:33:39.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 73454772-dc58-46ec-9840-9ab2a8682b05' 2024-05-15T12:33:39.933 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:33:39.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCjq0RmnfXmORAAc3N9E0ULcaU5haRIOuXzOg== 2024-05-15T12:33:39.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCjq0RmnfXmORAAc3N9E0ULcaU5haRIOuXzOg=="}' 2024-05-15T12:33:39.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 73454772-dc58-46ec-9840-9ab2a8682b05 -i td/test-erasure-code/2/new.json 2024-05-15T12:33:40.462 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:33:40.478 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/2/new.json 2024-05-15T12:33:40.479 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCjq0RmnfXmORAAc3N9E0ULcaU5haRIOuXzOg== --osd-uuid 73454772-dc58-46ec-9840-9ab2a8682b05 2024-05-15T12:33:40.507 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:40.517+0000 7f2fa85767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:33:40.517 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:40.529+0000 7f2fa85767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:33:40.518 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:40.529+0000 7f2fa85767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:33:40.519 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:40.533+0000 7f2fa85767c0 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-05-15T12:33:40.519 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:40.533+0000 7f2fa85767c0 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-05-15T12:33:40.519 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:40.533+0000 7f2fa85767c0 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-05-15T12:33:40.557 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:40.569+0000 7f2fa85767c0 -1 bluestore(td/test-erasure-code/2) _read_fsid unparsable uuid 2024-05-15T12:33:47.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/2/keyring 2024-05-15T12:33:47.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:33:47.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-15T12:33:47.521 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd2 key to auth repository 2024-05-15T12:33:47.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:33:48.078 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.2 2024-05-15T12:33:48.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-15T12:33:48.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:33:48.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:33:48.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:33:48.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:33:48.111 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:48.125+0000 7fd2ffd737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:33:48.131 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:48.145+0000 7fd2ffd737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:33:48.132 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:48.145+0000 7fd2ffd737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:33:48.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-15T12:33:48.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:33:48.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-15T12:33:48.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:33:48.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:33:48.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:48.507 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:33:48.508 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:33:48.508 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:48.508 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:33:48.706 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:48.716+0000 7fd2ffd737c0 -1 Falling back to public interface 2024-05-15T12:33:48.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:49.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:49.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:49.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:33:49.921 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:33:49.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:49.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:33:50.341 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:51.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:51.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:51.343 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:33:51.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:33:51.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:51.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:33:51.382 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:33:51.392+0000 7fd2ffd737c0 -1 osd.2 0 log_to_monitors true 2024-05-15T12:33:51.878 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:52.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:52.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:52.880 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:33:52.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:33:52.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:52.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:33:53.500 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:54.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:54.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:54.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:33:54.503 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:33:54.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:54.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:33:55.131 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:56.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:56.134 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:56.134 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:33:56.134 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:33:56.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:56.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:33:56.567 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:57.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:57.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:57.569 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:33:57.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:33:57.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:57.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:33:58.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:33:59.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:33:59.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:33:59.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:33:59.011 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:33:59.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:33:59.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:33:59.512 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:00.514 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:00.514 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:00.515 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:34:00.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:34:00.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:00.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:34:01.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:02.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:02.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:02.173 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:34:02.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:34:02.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:02.174 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:34:02.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:03.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:03.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:03.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:34:03.594 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:34:03.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:03.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:34:04.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:05.017 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:05.017 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:05.017 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:34:05.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:34:05.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:05.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:34:05.798 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:06.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:06.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:06.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:34:06.800 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:34:06.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:06.801 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:34:07.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:08.688 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:08.688 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:08.688 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:34:08.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:34:08.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:08.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:34:09.455 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:10.457 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:10.457 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:10.457 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:34:10.458 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:34:10.458 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:10.458 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:34:10.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:11.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:11.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:11.965 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:34:11.965 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:34:11.965 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:11.965 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:34:12.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:13.683 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:13.683 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:13.683 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:34:13.683 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:34:13.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:13.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:34:14.792 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:15.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:15.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:15.794 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:34:15.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:34:15.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:15.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:34:16.239 INFO:tasks.workunit.client.0.smithi005.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1198231982,v1:127.0.0.1:6819/1198231982] [v2:127.0.0.1:6820/1198231982,v1:127.0.0.1:6821/1198231982] exists,up 73454772-dc58-46ec-9840-9ab2a8682b05 2024-05-15T12:34:16.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:34:16.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:34:16.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:34:16.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-15T12:34:16.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 3 2024-05-15T12:34:16.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-15T12:34:16.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:34:16.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-15T12:34:16.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:34:16.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/3 2024-05-15T12:34:16.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-15T12:34:16.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:34:16.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:34:16.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:34:16.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2024-05-15T12:34:16.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2024-05-15T12:34:16.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:34:16.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:34:16.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-15T12:34:16.242 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:34:16.242 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:34:16.242 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:34:16.242 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:34:16.243 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:34:16.243 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:34:16.243 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:34:16.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:34:16.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:34:16.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:34:16.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:34:16.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-15T12:34:16.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-15T12:34:16.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:34:16.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:34:16.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:34:16.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:34:16.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:34:16.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:34:16.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/3 2024-05-15T12:34:16.244 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:34:16.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=26cfd0e8-10b6-4c44-988f-992a5fea5779 2024-05-15T12:34:16.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 26cfd0e8-10b6-4c44-988f-992a5fea5779' 2024-05-15T12:34:16.245 INFO:tasks.workunit.client.0.smithi005.stdout:add osd3 26cfd0e8-10b6-4c44-988f-992a5fea5779 2024-05-15T12:34:16.245 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:34:16.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDIq0RmUjaUEBAAVHCu+QTJKqRWw7TJERLrpQ== 2024-05-15T12:34:16.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDIq0RmUjaUEBAAVHCu+QTJKqRWw7TJERLrpQ=="}' 2024-05-15T12:34:16.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 26cfd0e8-10b6-4c44-988f-992a5fea5779 -i td/test-erasure-code/3/new.json 2024-05-15T12:34:16.876 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:34:16.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/3/new.json 2024-05-15T12:34:16.893 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDIq0RmUjaUEBAAVHCu+QTJKqRWw7TJERLrpQ== --osd-uuid 26cfd0e8-10b6-4c44-988f-992a5fea5779 2024-05-15T12:34:16.927 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:34:16.940+0000 7f728673e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:34:16.939 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:34:16.952+0000 7f728673e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:34:16.941 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:34:16.952+0000 7f728673e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:34:16.941 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:34:16.952+0000 7f728673e7c0 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-05-15T12:34:16.941 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:34:16.952+0000 7f728673e7c0 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-05-15T12:34:16.941 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:34:16.952+0000 7f728673e7c0 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-05-15T12:34:16.979 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:34:16.992+0000 7f728673e7c0 -1 bluestore(td/test-erasure-code/3) _read_fsid unparsable uuid 2024-05-15T12:34:23.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/3/keyring 2024-05-15T12:34:23.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:34:23.498 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd3 key to auth repository 2024-05-15T12:34:23.498 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-15T12:34:23.498 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:34:24.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-15T12:34:24.057 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.3 2024-05-15T12:34:24.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:34:24.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:34:24.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:34:24.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:34:24.090 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:34:24.100+0000 7efe122087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:34:24.112 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:34:24.124+0000 7efe122087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:34:24.113 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:34:24.124+0000 7efe122087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:34:24.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-15T12:34:24.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:34:24.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-15T12:34:24.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:34:24.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:34:24.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:24.472 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:34:24.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:34:24.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:24.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:34:24.676 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:34:24.688+0000 7efe122087c0 -1 Falling back to public interface 2024-05-15T12:34:24.894 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:25.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:25.896 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:25.896 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:34:25.896 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:34:25.896 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:25.896 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:34:26.316 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:27.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:27.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:27.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:34:27.318 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:34:27.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:27.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:34:27.336 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:34:27.348+0000 7efe122087c0 -1 osd.3 0 log_to_monitors true 2024-05-15T12:34:27.867 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:28.868 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:28.868 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:28.869 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:34:28.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:34:28.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:28.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:34:29.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:30.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:30.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:30.354 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:34:30.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:34:30.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:30.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:34:30.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:31.862 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:31.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:31.863 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:34:31.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:34:31.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:31.864 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:34:32.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:33.384 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:33.385 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:33.385 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:34:33.385 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:34:33.385 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:33.386 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:34:33.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:34.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:34.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:34.828 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:34:34.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:34:34.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:34.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:34:35.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:36.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:36.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:36.270 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:34:36.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:34:36.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:36.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:34:36.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:37.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:37.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:37.692 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:34:37.693 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:34:37.693 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:37.693 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:34:38.187 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:39.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:39.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:39.189 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:34:39.190 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:34:39.190 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:39.190 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:34:39.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:40.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:40.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:40.830 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:34:40.831 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:34:40.831 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:40.831 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:34:41.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:42.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:42.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:42.873 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:34:42.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:34:42.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:42.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:34:44.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:45.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:45.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:45.017 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:34:45.017 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:34:45.017 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:45.017 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:34:45.840 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:46.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:46.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:46.843 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:34:46.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:34:46.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:46.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:34:47.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:48.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:48.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:48.764 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:34:48.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:34:48.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:48.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:34:49.556 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:50.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:50.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:50.558 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:34:50.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:34:50.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:50.559 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:34:51.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:34:52.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:34:52.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:34:52.355 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:34:52.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:34:52.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:34:52.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:34:52.870 INFO:tasks.workunit.client.0.smithi005.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3089095356,v1:127.0.0.1:6827/3089095356] [v2:127.0.0.1:6828/3089095356,v1:127.0.0.1:6829/3089095356] exists,up 26cfd0e8-10b6-4c44-988f-992a5fea5779 2024-05-15T12:34:52.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:34:52.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:34:52.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:34:52.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-15T12:34:52.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 4 2024-05-15T12:34:52.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-15T12:34:52.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:34:52.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-05-15T12:34:52.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:34:52.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/4 2024-05-15T12:34:52.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-15T12:34:52.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:34:52.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:34:52.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:34:52.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/4' 2024-05-15T12:34:52.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/4/journal' 2024-05-15T12:34:52.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:34:52.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:34:52.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-15T12:34:52.874 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:34:52.874 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:34:52.874 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:34:52.874 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:34:52.874 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:34:52.875 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:34:52.875 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:34:52.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:34:52.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:34:52.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:34:52.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:34:52.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-15T12:34:52.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-15T12:34:52.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:34:52.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:34:52.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:34:52.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:34:52.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:34:52.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:34:52.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/4 2024-05-15T12:34:52.876 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:34:52.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5a590216-895d-46be-bad8-d28ef6a048b9 2024-05-15T12:34:52.876 INFO:tasks.workunit.client.0.smithi005.stdout:add osd4 5a590216-895d-46be-bad8-d28ef6a048b9 2024-05-15T12:34:52.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 5a590216-895d-46be-bad8-d28ef6a048b9' 2024-05-15T12:34:52.877 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:34:52.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDsq0Rml76LNhAAvWHfWbNmx+75h4KfJyZYug== 2024-05-15T12:34:52.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDsq0Rml76LNhAAvWHfWbNmx+75h4KfJyZYug=="}' 2024-05-15T12:34:52.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5a590216-895d-46be-bad8-d28ef6a048b9 -i td/test-erasure-code/4/new.json 2024-05-15T12:34:53.372 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:34:53.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/4/new.json 2024-05-15T12:34:53.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDsq0Rml76LNhAAvWHfWbNmx+75h4KfJyZYug== --osd-uuid 5a590216-895d-46be-bad8-d28ef6a048b9 2024-05-15T12:34:53.425 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:34:53.440+0000 7fbd118a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:34:53.434 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:34:53.448+0000 7fbd118a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:34:53.436 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:34:53.448+0000 7fbd118a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:34:53.436 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:34:53.448+0000 7fbd118a17c0 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-05-15T12:34:53.436 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:34:53.448+0000 7fbd118a17c0 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-05-15T12:34:53.436 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:34:53.448+0000 7fbd118a17c0 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-05-15T12:34:53.474 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:34:53.488+0000 7fbd118a17c0 -1 bluestore(td/test-erasure-code/4) _read_fsid unparsable uuid 2024-05-15T12:34:59.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/4/keyring 2024-05-15T12:34:59.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:34:59.957 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd4 key to auth repository 2024-05-15T12:34:59.958 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-05-15T12:34:59.958 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:35:00.522 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.4 2024-05-15T12:35:00.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-05-15T12:35:00.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:35:00.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:35:00.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:35:00.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:35:00.547 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:35:00.564+0000 7f6fa49507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:35:00.574 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:35:00.588+0000 7f6fa49507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:35:00.575 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:35:00.592+0000 7f6fa49507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:35:01.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-05-15T12:35:01.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:35:01.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-05-15T12:35:01.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:35:01.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:35:01.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:01.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:35:01.003 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:35:01.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:01.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:35:01.139 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:35:01.156+0000 7f6fa49507c0 -1 Falling back to public interface 2024-05-15T12:35:01.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:02.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:02.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:02.427 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:35:02.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:35:02.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:02.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:35:02.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:03.738 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:35:03.752+0000 7f6fa49507c0 -1 osd.4 0 log_to_monitors true 2024-05-15T12:35:03.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:03.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:03.854 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:35:03.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:35:03.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:03.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:35:04.404 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:05.406 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:05.406 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:05.406 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:35:05.407 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:35:05.407 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:05.407 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:35:05.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:06.958 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:06.958 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:06.958 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:35:06.958 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:35:06.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:06.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:35:07.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:08.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:08.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:08.446 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:35:08.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:35:08.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:08.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:35:08.990 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:09.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:09.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:09.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:35:09.995 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:35:09.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:09.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:35:10.609 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:11.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:11.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:11.611 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:35:11.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:35:11.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:11.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:35:12.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:13.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:13.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:13.053 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:35:13.053 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:35:13.053 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:13.053 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:35:13.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:14.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:14.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:14.567 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:35:14.567 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:35:14.567 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:14.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:35:15.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:16.162 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:16.162 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:16.162 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:35:16.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:35:16.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:16.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:35:16.702 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:17.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:17.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:17.704 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:35:17.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:35:17.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:17.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:35:18.510 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:19.511 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:19.512 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:19.512 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:35:19.512 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:35:19.512 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:19.512 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:35:20.212 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:21.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:21.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:21.214 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:35:21.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:35:21.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:21.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:35:22.151 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:23.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:23.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:23.153 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:35:23.154 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:35:23.154 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:23.154 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:35:23.905 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:24.907 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:24.907 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:24.907 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:35:24.908 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:35:24.908 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:24.908 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:35:26.053 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:27.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:27.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:27.055 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:35:27.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:35:27.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:27.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:35:27.952 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:28.954 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:28.954 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:28.954 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:35:28.954 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:35:28.955 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:28.955 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:35:29.379 INFO:tasks.workunit.client.0.smithi005.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/428542082,v1:127.0.0.1:6835/428542082] [v2:127.0.0.1:6836/428542082,v1:127.0.0.1:6837/428542082] exists,up 5a590216-895d-46be-bad8-d28ef6a048b9 2024-05-15T12:35:29.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:35:29.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:35:29.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:35:29.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-15T12:35:29.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 5 2024-05-15T12:35:29.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-15T12:35:29.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:35:29.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-05-15T12:35:29.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:35:29.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/5 2024-05-15T12:35:29.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-15T12:35:29.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:35:29.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:35:29.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:35:29.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2024-05-15T12:35:29.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2024-05-15T12:35:29.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:35:29.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:35:29.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-15T12:35:29.382 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:35:29.382 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:35:29.382 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:35:29.384 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:35:29.384 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:35:29.384 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:35:29.384 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:35:29.384 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:35:29.384 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:35:29.384 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:35:29.384 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:35:29.384 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-15T12:35:29.384 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-15T12:35:29.385 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:35:29.385 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:35:29.385 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:35:29.385 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:35:29.385 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:35:29.385 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:35:29.385 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/5 2024-05-15T12:35:29.385 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:35:29.386 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5330abbc-8264-44a7-a2a3-af755956921b 2024-05-15T12:35:29.386 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 5330abbc-8264-44a7-a2a3-af755956921b' 2024-05-15T12:35:29.386 INFO:tasks.workunit.client.0.smithi005.stdout:add osd5 5330abbc-8264-44a7-a2a3-af755956921b 2024-05-15T12:35:29.386 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:35:29.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQARrERmyNz9GBAAm9bD9B27Hj9OtCJvXfT0uw== 2024-05-15T12:35:29.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQARrERmyNz9GBAAm9bD9B27Hj9OtCJvXfT0uw=="}' 2024-05-15T12:35:29.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5330abbc-8264-44a7-a2a3-af755956921b -i td/test-erasure-code/5/new.json 2024-05-15T12:35:29.906 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:35:29.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/5/new.json 2024-05-15T12:35:29.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQARrERmyNz9GBAAm9bD9B27Hj9OtCJvXfT0uw== --osd-uuid 5330abbc-8264-44a7-a2a3-af755956921b 2024-05-15T12:35:29.956 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:35:29.971+0000 7f0b9789e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:35:29.987 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:35:30.003+0000 7f0b9789e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:35:29.989 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:35:30.003+0000 7f0b9789e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:35:29.990 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:35:30.003+0000 7f0b9789e7c0 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-05-15T12:35:29.990 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:35:30.003+0000 7f0b9789e7c0 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-05-15T12:35:29.990 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:35:30.003+0000 7f0b9789e7c0 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-05-15T12:35:30.027 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:35:30.043+0000 7f0b9789e7c0 -1 bluestore(td/test-erasure-code/5) _read_fsid unparsable uuid 2024-05-15T12:35:36.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/5/keyring 2024-05-15T12:35:36.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:35:36.961 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd5 key to auth repository 2024-05-15T12:35:36.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-05-15T12:35:36.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:35:37.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-05-15T12:35:37.519 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.5 2024-05-15T12:35:37.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:35:37.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:35:37.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:35:37.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:35:37.550 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:35:37.567+0000 7f4be94557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:35:37.562 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:35:37.579+0000 7f4be94557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:35:37.563 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:35:37.579+0000 7f4be94557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:35:37.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-05-15T12:35:37.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:35:37.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-05-15T12:35:37.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:35:37.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:35:37.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:37.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:35:37.931 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:35:37.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:37.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:35:38.143 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:35:38.159+0000 7f4be94557c0 -1 Falling back to public interface 2024-05-15T12:35:38.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:39.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:39.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:39.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:35:39.370 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:35:39.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:39.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:35:39.801 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:40.751 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:35:40.767+0000 7f4be94557c0 -1 osd.5 0 log_to_monitors true 2024-05-15T12:35:40.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:40.804 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:40.804 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:35:40.804 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:35:40.804 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:40.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:35:41.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:42.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:42.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:42.431 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:35:42.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:35:42.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:42.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:35:43.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:44.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:44.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:44.005 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:35:44.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:35:44.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:44.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:35:44.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:45.452 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:45.453 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:45.453 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:35:45.453 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:35:45.453 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:45.453 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:35:46.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:47.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:47.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:47.076 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:35:47.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:35:47.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:47.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:35:47.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:48.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:48.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:48.569 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:35:48.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:35:48.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:48.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:35:49.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:50.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:50.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:50.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:35:50.297 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:35:50.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:50.298 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:35:50.848 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:51.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:51.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:51.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:35:51.850 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:35:51.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:51.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:35:52.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:53.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:53.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:53.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:35:53.306 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:35:53.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:53.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:35:53.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:54.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:54.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:54.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:35:54.823 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:35:54.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:54.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:35:55.825 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:56.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:56.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:56.827 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:35:56.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:35:56.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:56.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:35:57.997 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:35:58.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:35:58.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:35:58.999 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:35:58.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:35:58.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:35:58.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:36:00.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:01.091 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:01.091 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:01.092 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:36:01.092 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:36:01.092 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:01.092 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:36:02.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:03.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:03.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:03.329 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:36:03.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:36:03.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:03.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:36:04.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:05.226 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:05.226 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:05.227 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:36:05.227 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:36:05.227 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:05.227 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:36:05.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:06.914 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:06.914 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:06.914 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:36:06.915 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:36:06.915 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:06.915 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:36:07.333 INFO:tasks.workunit.client.0.smithi005.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3080663749,v1:127.0.0.1:6843/3080663749] [v2:127.0.0.1:6844/3080663749,v1:127.0.0.1:6845/3080663749] exists,up 5330abbc-8264-44a7-a2a3-af755956921b 2024-05-15T12:36:07.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:36:07.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:36:07.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:36:07.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-15T12:36:07.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 6 2024-05-15T12:36:07.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-15T12:36:07.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:36:07.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-05-15T12:36:07.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:36:07.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/6 2024-05-15T12:36:07.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-15T12:36:07.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:36:07.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:36:07.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:36:07.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2024-05-15T12:36:07.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2024-05-15T12:36:07.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:36:07.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:36:07.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-15T12:36:07.337 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:36:07.337 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:36:07.337 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:36:07.338 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:36:07.339 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:36:07.339 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:36:07.339 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:36:07.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:36:07.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:36:07.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:36:07.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:36:07.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-15T12:36:07.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-15T12:36:07.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:36:07.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:36:07.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:36:07.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:36:07.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:36:07.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:36:07.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/6 2024-05-15T12:36:07.340 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:36:07.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7e2b0f2f-49ac-4c6a-860f-a361e974bd79 2024-05-15T12:36:07.341 INFO:tasks.workunit.client.0.smithi005.stdout:add osd6 7e2b0f2f-49ac-4c6a-860f-a361e974bd79 2024-05-15T12:36:07.341 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 7e2b0f2f-49ac-4c6a-860f-a361e974bd79' 2024-05-15T12:36:07.341 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:36:07.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA3rERmYAdKFhAAluIIDINcbJrxUNhzXe9D7w== 2024-05-15T12:36:07.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA3rERmYAdKFhAAluIIDINcbJrxUNhzXe9D7w=="}' 2024-05-15T12:36:07.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7e2b0f2f-49ac-4c6a-860f-a361e974bd79 -i td/test-erasure-code/6/new.json 2024-05-15T12:36:07.882 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:36:07.900 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/6/new.json 2024-05-15T12:36:07.901 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA3rERmYAdKFhAAluIIDINcbJrxUNhzXe9D7w== --osd-uuid 7e2b0f2f-49ac-4c6a-860f-a361e974bd79 2024-05-15T12:36:07.936 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:07.951+0000 7feaf93ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:36:07.942 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:07.959+0000 7feaf93ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:36:07.944 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:07.959+0000 7feaf93ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:36:07.944 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:07.959+0000 7feaf93ce7c0 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-05-15T12:36:07.944 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:07.959+0000 7feaf93ce7c0 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-05-15T12:36:07.944 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:07.959+0000 7feaf93ce7c0 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-05-15T12:36:07.990 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:08.007+0000 7feaf93ce7c0 -1 bluestore(td/test-erasure-code/6) _read_fsid unparsable uuid 2024-05-15T12:36:14.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/6/keyring 2024-05-15T12:36:14.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:36:14.276 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd6 key to auth repository 2024-05-15T12:36:14.276 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-05-15T12:36:14.276 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:36:14.848 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.6 2024-05-15T12:36:14.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-05-15T12:36:14.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:36:14.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:36:14.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:36:14.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:36:14.880 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:14.895+0000 7fea116d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:36:14.892 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:14.907+0000 7fea116d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:36:14.893 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:14.907+0000 7fea116d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:36:15.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-05-15T12:36:15.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:36:15.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-05-15T12:36:15.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:36:15.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:36:15.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:15.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:36:15.266 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:36:15.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:15.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:36:15.465 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:15.479+0000 7fea116d97c0 -1 Falling back to public interface 2024-05-15T12:36:15.688 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:16.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:16.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:16.691 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:36:16.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:36:16.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:16.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:36:17.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:18.042 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:18.059+0000 7fea116d97c0 -1 osd.6 0 log_to_monitors true 2024-05-15T12:36:18.152 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:18.152 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:18.152 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:36:18.152 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:36:18.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:18.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:36:18.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:19.878 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:19.878 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:19.878 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:36:19.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:36:19.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:19.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:36:20.326 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:21.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:21.328 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:36:21.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:21.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:36:21.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:21.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:36:21.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:22.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:22.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:22.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:36:22.816 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:36:22.817 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:22.817 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:36:23.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:24.525 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:24.525 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:24.525 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:36:24.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:36:24.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:24.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:36:25.088 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:26.089 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:26.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:26.090 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:36:26.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:36:26.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:26.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:36:26.530 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:27.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:27.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:27.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:36:27.532 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:36:27.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:27.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:36:28.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:29.041 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:29.041 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:29.041 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:36:29.041 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:36:29.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:29.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:36:29.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:30.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:30.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:30.528 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:36:30.528 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:36:30.528 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:30.528 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:36:31.044 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:32.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:32.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:32.046 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:36:32.046 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:36:32.046 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:32.046 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:36:32.891 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:33.893 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:33.893 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:33.893 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:36:33.893 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:36:33.894 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:33.894 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:36:34.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:35.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:35.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:35.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:36:35.661 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:36:35.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:35.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:36:36.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:37.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:37.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:37.494 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:36:37.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:36:37.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:37.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:36:38.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:39.596 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:39.596 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:39.596 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:36:39.597 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:36:39.597 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:39.597 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:36:40.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:41.322 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:41.322 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:41.323 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:36:41.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:36:41.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:41.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:36:42.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:43.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:43.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:43.333 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:36:43.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:36:43.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:43.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:36:43.756 INFO:tasks.workunit.client.0.smithi005.stdout:osd.6 up in weight 1 up_from 34 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1555829247,v1:127.0.0.1:6851/1555829247] [v2:127.0.0.1:6852/1555829247,v1:127.0.0.1:6853/1555829247] exists,up 7e2b0f2f-49ac-4c6a-860f-a361e974bd79 2024-05-15T12:36:43.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:36:43.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:36:43.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:36:43.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-15T12:36:43.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 7 2024-05-15T12:36:43.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-15T12:36:43.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:36:43.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-05-15T12:36:43.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:36:43.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/7 2024-05-15T12:36:43.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-15T12:36:43.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:36:43.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:36:43.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:36:43.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/7' 2024-05-15T12:36:43.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/7/journal' 2024-05-15T12:36:43.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:36:43.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:36:43.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-15T12:36:43.762 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:36:43.762 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:36:43.762 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:36:43.762 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:36:43.762 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:36:43.762 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:36:43.762 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:36:43.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:36:43.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:36:43.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:36:43.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:36:43.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-15T12:36:43.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-15T12:36:43.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:36:43.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:36:43.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:36:43.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:36:43.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:36:43.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:36:43.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/7 2024-05-15T12:36:43.763 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:36:43.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9c3ac2fb-7cf6-4fe7-991b-b5f1f1e32623 2024-05-15T12:36:43.764 INFO:tasks.workunit.client.0.smithi005.stdout:add osd7 9c3ac2fb-7cf6-4fe7-991b-b5f1f1e32623 2024-05-15T12:36:43.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 9c3ac2fb-7cf6-4fe7-991b-b5f1f1e32623' 2024-05-15T12:36:43.764 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:36:43.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBbrERmQ+RjLxAADaBDmQSecSowb0+wis2j6Q== 2024-05-15T12:36:43.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBbrERmQ+RjLxAADaBDmQSecSowb0+wis2j6Q=="}' 2024-05-15T12:36:43.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9c3ac2fb-7cf6-4fe7-991b-b5f1f1e32623 -i td/test-erasure-code/7/new.json 2024-05-15T12:36:44.263 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:36:44.278 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/7/new.json 2024-05-15T12:36:44.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBbrERmQ+RjLxAADaBDmQSecSowb0+wis2j6Q== --osd-uuid 9c3ac2fb-7cf6-4fe7-991b-b5f1f1e32623 2024-05-15T12:36:44.313 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:44.326+0000 7fb40a0e17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:36:44.495 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:44.510+0000 7fb40a0e17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:36:44.496 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:44.510+0000 7fb40a0e17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:36:44.497 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:44.510+0000 7fb40a0e17c0 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-05-15T12:36:44.497 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:44.510+0000 7fb40a0e17c0 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-05-15T12:36:44.497 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:44.510+0000 7fb40a0e17c0 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-05-15T12:36:44.569 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:44.582+0000 7fb40a0e17c0 -1 bluestore(td/test-erasure-code/7) _read_fsid unparsable uuid 2024-05-15T12:36:51.968 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/7/keyring 2024-05-15T12:36:51.968 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:36:51.969 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd7 key to auth repository 2024-05-15T12:36:51.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-05-15T12:36:51.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:36:52.525 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.7 2024-05-15T12:36:52.525 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-05-15T12:36:52.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:36:52.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:36:52.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:36:52.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:36:52.557 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:52.570+0000 7fbe1cef67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:36:52.559 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:52.574+0000 7fbe1cef67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:36:52.561 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:52.574+0000 7fbe1cef67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:36:53.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-05-15T12:36:53.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:36:53.038 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-05-15T12:36:53.038 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:36:53.038 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:36:53.038 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:53.039 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:36:53.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:36:53.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:53.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:36:53.131 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:53.146+0000 7fbe1cef67c0 -1 Falling back to public interface 2024-05-15T12:36:53.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:54.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:54.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:54.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:36:54.488 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:36:54.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:54.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:36:54.954 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:55.757 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:36:55.770+0000 7fbe1cef67c0 -1 osd.7 0 log_to_monitors true 2024-05-15T12:36:55.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:55.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:55.956 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:36:55.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:36:55.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:55.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:36:56.552 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:57.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:57.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:57.554 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:36:57.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:36:57.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:57.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:36:58.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:36:59.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:36:59.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:36:59.195 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:36:59.196 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:36:59.196 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:36:59.196 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:36:59.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:00.737 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:00.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:00.738 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:37:00.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:37:00.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:00.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:37:01.207 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:02.209 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:02.209 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:02.209 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:37:02.210 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:37:02.210 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:02.210 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:37:02.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:03.688 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:03.688 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:03.688 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:37:03.688 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:37:03.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:03.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:37:04.180 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:05.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:05.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:05.182 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:37:05.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:37:05.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:05.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:37:05.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:06.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:06.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:06.710 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:37:06.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:37:06.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:06.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:37:07.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:08.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:08.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:08.286 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:37:08.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:37:08.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:08.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:37:08.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:09.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:09.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:09.805 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:37:09.806 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:37:09.806 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:09.806 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:37:10.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:11.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:11.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:11.236 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:37:11.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:37:11.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:11.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:37:11.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:12.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:12.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:12.939 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:37:12.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:37:12.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:12.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:37:13.897 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:14.898 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:14.898 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:14.899 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:37:14.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:37:14.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:14.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:37:16.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:17.010 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:17.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:17.011 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:37:17.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:37:17.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:17.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:37:17.909 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:18.911 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:18.911 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:18.911 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:37:18.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:37:18.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:18.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:37:20.208 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:21.210 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:21.210 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:21.211 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:37:21.211 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:37:21.211 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:21.211 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:37:22.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:23.203 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:23.203 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:23.203 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T12:37:23.204 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T12:37:23.204 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:23.204 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:37:23.685 INFO:tasks.workunit.client.0.smithi005.stdout:osd.7 up in weight 1 up_from 39 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/2041948639,v1:127.0.0.1:6859/2041948639] [v2:127.0.0.1:6860/2041948639,v1:127.0.0.1:6861/2041948639] exists,up 9c3ac2fb-7cf6-4fe7-991b-b5f1f1e32623 2024-05-15T12:37:23.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:37:23.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:37:23.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:37:23.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-15T12:37:23.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 8 2024-05-15T12:37:23.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-15T12:37:23.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:37:23.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-05-15T12:37:23.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:37:23.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/8 2024-05-15T12:37:23.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-15T12:37:23.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:37:23.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:37:23.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:37:23.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/8' 2024-05-15T12:37:23.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/8/journal' 2024-05-15T12:37:23.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:37:23.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:37:23.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-15T12:37:23.690 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:37:23.690 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:37:23.690 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:37:23.691 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:37:23.691 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:37:23.691 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:37:23.691 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:37:23.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:37:23.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:37:23.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:37:23.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:37:23.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-15T12:37:23.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-15T12:37:23.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:37:23.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:37:23.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:37:23.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:37:23.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:37:23.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:37:23.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/8 2024-05-15T12:37:23.692 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:37:23.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=770a8dbf-ed13-4feb-88f4-5311f474c5a2 2024-05-15T12:37:23.693 INFO:tasks.workunit.client.0.smithi005.stdout:add osd8 770a8dbf-ed13-4feb-88f4-5311f474c5a2 2024-05-15T12:37:23.693 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 770a8dbf-ed13-4feb-88f4-5311f474c5a2' 2024-05-15T12:37:23.693 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:37:23.713 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCDrERm9NyAKxAAj6Ce78sThzWG2tfVBkoRZA== 2024-05-15T12:37:23.713 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCDrERm9NyAKxAAj6Ce78sThzWG2tfVBkoRZA=="}' 2024-05-15T12:37:23.713 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 770a8dbf-ed13-4feb-88f4-5311f474c5a2 -i td/test-erasure-code/8/new.json 2024-05-15T12:37:24.195 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:37:24.210 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/8/new.json 2024-05-15T12:37:24.211 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCDrERm9NyAKxAAj6Ce78sThzWG2tfVBkoRZA== --osd-uuid 770a8dbf-ed13-4feb-88f4-5311f474c5a2 2024-05-15T12:37:24.238 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:37:24.249+0000 7f1a311af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:37:24.279 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:37:24.293+0000 7f1a311af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:37:24.281 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:37:24.293+0000 7f1a311af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:37:24.281 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:37:24.293+0000 7f1a311af7c0 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-05-15T12:37:24.281 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:37:24.293+0000 7f1a311af7c0 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-05-15T12:37:24.281 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:37:24.293+0000 7f1a311af7c0 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-05-15T12:37:24.334 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:37:24.345+0000 7f1a311af7c0 -1 bluestore(td/test-erasure-code/8) _read_fsid unparsable uuid 2024-05-15T12:37:30.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/8/keyring 2024-05-15T12:37:30.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:37:30.707 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd8 key to auth repository 2024-05-15T12:37:30.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-05-15T12:37:30.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:37:31.265 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.8 2024-05-15T12:37:31.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-05-15T12:37:31.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:37:31.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:37:31.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:37:31.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:37:31.297 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:37:31.309+0000 7fb37106a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:37:31.299 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:37:31.313+0000 7fb37106a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:37:31.300 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:37:31.313+0000 7fb37106a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:37:31.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-05-15T12:37:31.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:37:31.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-05-15T12:37:31.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:37:31.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:37:31.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:31.728 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:37:31.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:37:31.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:31.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:37:31.848 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:37:31.861+0000 7fb37106a7c0 -1 Falling back to public interface 2024-05-15T12:37:32.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:33.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:33.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:33.184 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:37:33.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:37:33.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:33.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:37:33.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:34.401 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:37:34.413+0000 7fb37106a7c0 -1 osd.8 0 log_to_monitors true 2024-05-15T12:37:34.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:34.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:34.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:37:34.607 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:37:34.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:34.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:37:35.162 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:36.164 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:36.164 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:36.164 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:37:36.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:37:36.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:36.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:37:36.714 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:37.716 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:37.716 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:37.716 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:37:37.717 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:37:37.717 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:37.717 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:37:38.302 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:39.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:39.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:39.304 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:37:39.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:37:39.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:39.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:37:39.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:40.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:40.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:40.854 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:37:40.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:37:40.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:40.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:37:41.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:42.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:42.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:42.346 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:37:42.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:37:42.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:42.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:37:42.906 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:43.908 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:43.908 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:43.908 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:37:43.909 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:37:43.909 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:43.909 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:37:44.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:45.389 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:45.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:45.390 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:37:45.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:37:45.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:45.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:37:45.867 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:46.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:46.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:46.869 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:37:46.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:37:46.870 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:46.870 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:37:47.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:48.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:48.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:48.520 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:37:48.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:37:48.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:48.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:37:50.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:51.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:51.134 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:51.134 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:37:51.134 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:37:51.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:51.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:37:52.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:53.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:53.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:53.118 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:37:53.118 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:37:53.118 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:53.118 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:37:53.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:54.886 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:54.886 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:54.887 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:37:54.887 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:37:54.887 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:54.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:37:55.841 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:56.842 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:56.843 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:37:56.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:56.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:37:56.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:56.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:37:57.512 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:37:58.514 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:37:58.514 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:37:58.514 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:37:58.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:37:58.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:37:58.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:37:59.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:00.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:00.028 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:00.028 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:38:00.028 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:38:00.029 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:00.029 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:38:00.468 INFO:tasks.workunit.client.0.smithi005.stdout:osd.8 up in weight 1 up_from 44 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/276341482,v1:127.0.0.1:6867/276341482] [v2:127.0.0.1:6868/276341482,v1:127.0.0.1:6869/276341482] exists,up 770a8dbf-ed13-4feb-88f4-5311f474c5a2 2024-05-15T12:38:00.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:38:00.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:38:00.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:38:00.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-15T12:38:00.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 9 2024-05-15T12:38:00.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-15T12:38:00.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:38:00.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-05-15T12:38:00.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:38:00.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/9 2024-05-15T12:38:00.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-15T12:38:00.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:38:00.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:38:00.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:38:00.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2024-05-15T12:38:00.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2024-05-15T12:38:00.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:38:00.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:38:00.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-15T12:38:00.472 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:38:00.473 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:38:00.473 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:38:00.473 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:38:00.473 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:38:00.473 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:38:00.473 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:38:00.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:38:00.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:38:00.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:38:00.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:38:00.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-15T12:38:00.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-15T12:38:00.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:38:00.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:38:00.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:38:00.474 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:38:00.474 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:38:00.474 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:38:00.474 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/9 2024-05-15T12:38:00.474 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:38:00.474 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fb772ddb-911d-4848-a049-c1fce0d43178 2024-05-15T12:38:00.474 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 fb772ddb-911d-4848-a049-c1fce0d43178' 2024-05-15T12:38:00.475 INFO:tasks.workunit.client.0.smithi005.stdout:add osd9 fb772ddb-911d-4848-a049-c1fce0d43178 2024-05-15T12:38:00.475 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:38:00.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCorERmVt+BHhAAKV2e4mTtt0XKXETVGfzTNA== 2024-05-15T12:38:00.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCorERmVt+BHhAAKV2e4mTtt0XKXETVGfzTNA=="}' 2024-05-15T12:38:00.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fb772ddb-911d-4848-a049-c1fce0d43178 -i td/test-erasure-code/9/new.json 2024-05-15T12:38:01.074 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:38:01.094 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/9/new.json 2024-05-15T12:38:01.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCorERmVt+BHhAAKV2e4mTtt0XKXETVGfzTNA== --osd-uuid fb772ddb-911d-4848-a049-c1fce0d43178 2024-05-15T12:38:01.129 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:01.141+0000 7efc5235b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:38:01.153 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:01.165+0000 7efc5235b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:38:01.155 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:01.169+0000 7efc5235b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:38:01.155 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:01.169+0000 7efc5235b7c0 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-05-15T12:38:01.155 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:01.169+0000 7efc5235b7c0 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-05-15T12:38:01.155 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:01.169+0000 7efc5235b7c0 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-05-15T12:38:01.245 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:01.257+0000 7efc5235b7c0 -1 bluestore(td/test-erasure-code/9) _read_fsid unparsable uuid 2024-05-15T12:38:07.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/9/keyring 2024-05-15T12:38:07.968 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:38:07.969 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd9 key to auth repository 2024-05-15T12:38:07.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-05-15T12:38:07.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:38:08.541 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.9 2024-05-15T12:38:08.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-05-15T12:38:08.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:38:08.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:38:08.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:38:08.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:38:08.573 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:08.584+0000 7f5bb54997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:38:08.575 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:08.588+0000 7f5bb54997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:38:08.577 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:08.588+0000 7f5bb54997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:38:09.030 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-05-15T12:38:09.030 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:38:09.030 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-05-15T12:38:09.030 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:38:09.030 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:38:09.031 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:09.031 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:38:09.031 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:38:09.031 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:09.031 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:38:09.121 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:09.132+0000 7f5bb54997c0 -1 Falling back to public interface 2024-05-15T12:38:09.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:10.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:10.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:10.526 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:38:10.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:38:10.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:10.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:38:10.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:11.796 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:11.808+0000 7f5bb54997c0 -1 osd.9 0 log_to_monitors true 2024-05-15T12:38:11.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:11.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:11.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:38:11.960 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:38:11.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:11.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:38:12.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:13.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:13.528 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:13.528 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:38:13.528 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:38:13.528 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:13.528 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:38:14.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:15.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:15.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:15.061 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:38:15.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:38:15.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:15.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:38:15.616 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:16.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:16.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:16.618 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:38:16.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:38:16.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:16.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:38:17.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:18.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:18.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:18.258 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:38:18.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:38:18.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:18.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:38:18.840 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:19.841 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:19.841 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:19.842 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:38:19.842 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:38:19.842 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:19.842 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:38:20.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:21.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:21.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:21.327 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:38:21.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:38:21.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:21.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:38:21.798 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:22.799 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:22.799 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:22.800 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:38:22.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:38:22.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:22.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:38:23.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:24.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:24.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:24.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:38:24.354 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:38:24.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:24.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:38:24.782 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:25.784 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:25.784 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:25.784 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:38:25.785 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:38:25.785 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:25.785 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:38:26.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:27.238 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:27.238 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:27.239 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:38:27.239 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:38:27.239 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:27.239 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:38:28.345 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:29.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:29.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:29.348 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:38:29.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:38:29.349 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:29.349 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:38:29.861 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:30.862 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:30.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:30.863 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:38:30.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:38:30.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:30.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:38:31.942 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:32.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:32.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:32.944 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:38:32.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:38:32.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:32.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:38:33.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:34.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:34.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:34.670 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:38:34.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:38:34.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:34.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:38:35.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:36.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:36.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:36.400 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:38:36.401 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:38:36.401 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:36.401 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:38:36.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:37.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:37.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:37.961 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T12:38:37.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T12:38:37.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:37.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:38:38.418 INFO:tasks.workunit.client.0.smithi005.stdout:osd.9 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/1797512247,v1:127.0.0.1:6875/1797512247] [v2:127.0.0.1:6876/1797512247,v1:127.0.0.1:6877/1797512247] exists,up fb772ddb-911d-4848-a049-c1fce0d43178 2024-05-15T12:38:38.421 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:38:38.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:38:38.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:38:38.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-15T12:38:38.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 10 2024-05-15T12:38:38.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-15T12:38:38.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:38:38.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=10 2024-05-15T12:38:38.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:38:38.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/10 2024-05-15T12:38:38.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-15T12:38:38.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:38:38.423 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:38:38.423 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:38:38.423 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/10' 2024-05-15T12:38:38.423 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/10/journal' 2024-05-15T12:38:38.423 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:38:38.423 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:38:38.423 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-15T12:38:38.423 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:38:38.423 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:38:38.423 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:38:38.423 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:38:38.423 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:38:38.423 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:38:38.423 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:38:38.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:38:38.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:38:38.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:38:38.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:38:38.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-15T12:38:38.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-15T12:38:38.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:38:38.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:38:38.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:38:38.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:38:38.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:38:38.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:38:38.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/10 2024-05-15T12:38:38.425 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:38:38.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f2575b72-cedf-491f-8bc7-303eadfd04eb 2024-05-15T12:38:38.425 INFO:tasks.workunit.client.0.smithi005.stdout:add osd10 f2575b72-cedf-491f-8bc7-303eadfd04eb 2024-05-15T12:38:38.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd10 f2575b72-cedf-491f-8bc7-303eadfd04eb' 2024-05-15T12:38:38.426 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:38:38.441 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDOrERmZGlGGxAABXCEcIjTXT1Izq9Duv/IoA== 2024-05-15T12:38:38.441 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDOrERmZGlGGxAABXCEcIjTXT1Izq9Duv/IoA=="}' 2024-05-15T12:38:38.441 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f2575b72-cedf-491f-8bc7-303eadfd04eb -i td/test-erasure-code/10/new.json 2024-05-15T12:38:39.013 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:38:39.032 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/10/new.json 2024-05-15T12:38:39.033 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 10 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDOrERmZGlGGxAABXCEcIjTXT1Izq9Duv/IoA== --osd-uuid f2575b72-cedf-491f-8bc7-303eadfd04eb 2024-05-15T12:38:39.069 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:39.084+0000 7fbf9e3a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:38:39.084 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:39.100+0000 7fbf9e3a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:38:39.086 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:39.100+0000 7fbf9e3a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:38:39.086 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:39.100+0000 7fbf9e3a07c0 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-05-15T12:38:39.086 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:39.100+0000 7fbf9e3a07c0 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-05-15T12:38:39.086 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:39.100+0000 7fbf9e3a07c0 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-05-15T12:38:39.239 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:39.256+0000 7fbf9e3a07c0 -1 bluestore(td/test-erasure-code/10) _read_fsid unparsable uuid 2024-05-15T12:38:45.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/10/keyring 2024-05-15T12:38:45.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:38:45.981 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd10 key to auth repository 2024-05-15T12:38:45.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd10 key to auth repository 2024-05-15T12:38:45.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/10/keyring auth add osd.10 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:38:46.547 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.10 2024-05-15T12:38:46.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.10 2024-05-15T12:38:46.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 10 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:38:46.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:38:46.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:38:46.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:38:46.579 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:46.595+0000 7fb3aafca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:38:46.597 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:46.611+0000 7fb3aafca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:38:46.598 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:46.611+0000 7fb3aafca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:38:47.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 10 2024-05-15T12:38:47.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:38:47.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=10 2024-05-15T12:38:47.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:38:47.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:38:47.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:47.004 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:38:47.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:38:47.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:47.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-15T12:38:47.172 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:47.187+0000 7fb3aafca7c0 -1 Falling back to public interface 2024-05-15T12:38:47.437 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:48.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:48.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:48.439 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:38:48.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:38:48.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:48.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-15T12:38:48.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:49.813 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:38:49.827+0000 7fb3aafca7c0 -1 osd.10 0 log_to_monitors true 2024-05-15T12:38:49.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:49.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:49.884 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:38:49.885 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:38:49.885 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:49.885 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-15T12:38:50.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:51.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:51.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:51.638 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:38:51.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:38:51.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:51.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-15T12:38:52.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:53.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:53.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:53.229 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:38:53.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:38:53.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:53.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-15T12:38:53.726 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:54.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:54.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:54.728 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:38:54.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:38:54.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:54.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-15T12:38:55.220 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:56.222 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:56.222 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:56.222 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:38:56.223 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:38:56.223 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:56.223 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-15T12:38:56.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:57.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:57.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:57.694 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:38:57.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:38:57.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:57.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-15T12:38:58.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:38:59.131 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:38:59.131 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:38:59.131 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:38:59.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:38:59.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:38:59.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-15T12:38:59.629 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:39:00.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:39:00.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:39:00.631 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:39:00.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:39:00.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:39:00.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-15T12:39:01.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:39:02.051 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:39:02.051 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:39:02.052 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:39:02.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:39:02.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:39:02.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-15T12:39:02.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:39:03.562 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:39:03.562 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:39:03.563 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:39:03.563 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:39:03.563 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:39:03.563 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-15T12:39:04.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:39:05.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:39:05.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:39:05.215 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:39:05.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:39:05.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:39:05.216 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-15T12:39:06.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:39:07.423 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:39:07.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:39:07.424 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:39:07.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:39:07.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:39:07.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-15T12:39:08.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:39:09.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:39:09.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:39:09.264 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:39:09.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:39:09.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:39:09.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-15T12:39:09.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:39:10.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:39:10.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:39:10.815 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:39:10.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:39:10.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:39:10.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-15T12:39:11.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:39:12.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:39:12.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:39:12.281 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:39:12.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:39:12.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:39:12.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-15T12:39:13.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:39:14.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:39:14.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:39:14.427 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:39:14.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:39:14.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:39:14.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-15T12:39:14.878 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:39:15.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:39:15.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:39:15.880 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T12:39:15.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T12:39:15.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:39:15.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-15T12:39:16.341 INFO:tasks.workunit.client.0.smithi005.stdout:osd.10 up in weight 1 up_from 54 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6882/3317551408,v1:127.0.0.1:6883/3317551408] [v2:127.0.0.1:6884/3317551408,v1:127.0.0.1:6885/3317551408] exists,up f2575b72-cedf-491f-8bc7-303eadfd04eb 2024-05-15T12:39:16.342 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:39:16.342 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:39:16.342 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:39:16.342 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:39: run: create_rbd_pool 2024-05-15T12:39:16.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-15T12:39:16.786 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd' does not exist 2024-05-15T12:39:16.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-15T12:39:16.804 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-15T12:39:17.516 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd' already exists 2024-05-15T12:39:17.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T12:39:18.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-15T12:39:22.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:40: run: wait_for_clean 2024-05-15T12:39:22.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T12:39:22.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T12:39:22.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T12:39:22.258 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T12:39:22.259 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T12:39:22.259 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T12:39:22.260 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T12:39:22.260 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T12:39:22.260 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T12:39:22.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T12:39:22.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T12:39:22.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T12:39:22.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T12:39:22.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T12:39:22.381 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T12:39:22.867 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T12:39:22.867 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:39:22.867 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T12:39:22.867 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:39:22.867 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T12:39:22.867 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T12:39:22.868 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T12:39:22.868 INFO:tasks.workunit.client.0.smithi005.stderr:7 2024-05-15T12:39:22.868 INFO:tasks.workunit.client.0.smithi005.stderr:8 2024-05-15T12:39:22.868 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:39:22.868 INFO:tasks.workunit.client.0.smithi005.stderr:10' 2024-05-15T12:39:22.868 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T12:39:22.868 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:22.868 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T12:39:23.071 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836561 2024-05-15T12:39:23.071 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836561 2024-05-15T12:39:23.071 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561' 2024-05-15T12:39:23.071 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:23.071 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T12:39:23.231 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673036 2024-05-15T12:39:23.231 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673036 2024-05-15T12:39:23.231 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673036' 2024-05-15T12:39:23.231 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:23.231 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T12:39:23.384 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509507 2024-05-15T12:39:23.384 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509507 2024-05-15T12:39:23.384 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673036 2-64424509507' 2024-05-15T12:39:23.384 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:23.385 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T12:39:23.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345980 2024-05-15T12:39:23.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345980 2024-05-15T12:39:23.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673036 2-64424509507 3-85899345980' 2024-05-15T12:39:23.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:23.543 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T12:39:23.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182453 2024-05-15T12:39:23.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182453 2024-05-15T12:39:23.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673036 2-64424509507 3-85899345980 4-107374182453' 2024-05-15T12:39:23.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:23.705 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-15T12:39:23.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051630 2024-05-15T12:39:23.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051630 2024-05-15T12:39:23.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673036 2-64424509507 3-85899345980 4-107374182453 5-124554051630' 2024-05-15T12:39:23.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:23.860 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-15T12:39:24.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888106 2024-05-15T12:39:24.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888106 2024-05-15T12:39:24.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673036 2-64424509507 3-85899345980 4-107374182453 5-124554051630 6-146028888106' 2024-05-15T12:39:24.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:24.016 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-15T12:39:24.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724578 2024-05-15T12:39:24.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724578 2024-05-15T12:39:24.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673036 2-64424509507 3-85899345980 4-107374182453 5-124554051630 6-146028888106 7-167503724578' 2024-05-15T12:39:24.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:24.176 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-15T12:39:24.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561047 2024-05-15T12:39:24.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561047 2024-05-15T12:39:24.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673036 2-64424509507 3-85899345980 4-107374182453 5-124554051630 6-146028888106 7-167503724578 8-188978561047' 2024-05-15T12:39:24.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:24.327 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-05-15T12:39:24.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397522 2024-05-15T12:39:24.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397522 2024-05-15T12:39:24.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673036 2-64424509507 3-85899345980 4-107374182453 5-124554051630 6-146028888106 7-167503724578 8-188978561047 9-210453397522' 2024-05-15T12:39:24.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:24.481 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-05-15T12:39:24.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928233990 2024-05-15T12:39:24.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928233990 2024-05-15T12:39:24.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673036 2-64424509507 3-85899345980 4-107374182453 5-124554051630 6-146028888106 7-167503724578 8-188978561047 9-210453397522 10-231928233990' 2024-05-15T12:39:24.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:24.636 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836561 2024-05-15T12:39:24.636 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:24.637 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T12:39:24.638 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836561 2024-05-15T12:39:24.638 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:24.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836561 2024-05-15T12:39:24.639 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 21474836561 2024-05-15T12:39:24.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836561' 2024-05-15T12:39:24.640 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T12:39:25.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836561 -lt 21474836561 2024-05-15T12:39:25.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:25.077 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673036 2024-05-15T12:39:25.077 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:25.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T12:39:25.080 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673036 2024-05-15T12:39:25.080 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:25.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673036 2024-05-15T12:39:25.081 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 42949673036 2024-05-15T12:39:25.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673036' 2024-05-15T12:39:25.082 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T12:39:25.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673036 -lt 42949673036 2024-05-15T12:39:25.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:25.487 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509507 2024-05-15T12:39:25.487 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:25.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T12:39:25.489 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509507 2024-05-15T12:39:25.490 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:25.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509507 2024-05-15T12:39:25.491 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 64424509507 2024-05-15T12:39:25.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509507' 2024-05-15T12:39:25.491 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T12:39:25.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509507 -lt 64424509507 2024-05-15T12:39:25.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:25.963 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345980 2024-05-15T12:39:25.963 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:25.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T12:39:25.965 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345980 2024-05-15T12:39:25.965 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:25.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345980 2024-05-15T12:39:25.967 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 85899345980 2024-05-15T12:39:25.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345980' 2024-05-15T12:39:25.968 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T12:39:26.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345980 -lt 85899345980 2024-05-15T12:39:26.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:26.440 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182453 2024-05-15T12:39:26.440 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:26.442 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T12:39:26.443 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182453 2024-05-15T12:39:26.443 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:26.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182453 2024-05-15T12:39:26.444 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.4 seq 107374182453 2024-05-15T12:39:26.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182453' 2024-05-15T12:39:26.445 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T12:39:26.902 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182454 -lt 107374182453 2024-05-15T12:39:26.902 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:26.903 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-124554051630 2024-05-15T12:39:26.904 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:26.905 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-15T12:39:26.906 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-124554051630 2024-05-15T12:39:26.906 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:26.907 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051630 2024-05-15T12:39:26.908 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 124554051630' 2024-05-15T12:39:26.908 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.5 seq 124554051630 2024-05-15T12:39:26.908 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-15T12:39:27.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051631 -lt 124554051630 2024-05-15T12:39:27.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:27.378 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:27.378 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888106 2024-05-15T12:39:27.379 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-15T12:39:27.380 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:27.380 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888106 2024-05-15T12:39:27.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888106 2024-05-15T12:39:27.381 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.6 seq 146028888106 2024-05-15T12:39:27.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888106' 2024-05-15T12:39:27.382 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-15T12:39:27.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888106 -lt 146028888106 2024-05-15T12:39:27.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:27.813 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724578 2024-05-15T12:39:27.813 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:27.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-05-15T12:39:27.815 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724578 2024-05-15T12:39:27.815 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:27.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724578 2024-05-15T12:39:27.816 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.7 seq 167503724578 2024-05-15T12:39:27.817 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724578' 2024-05-15T12:39:27.817 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-05-15T12:39:28.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724578 -lt 167503724578 2024-05-15T12:39:28.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:28.440 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561047 2024-05-15T12:39:28.440 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:28.442 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-15T12:39:28.443 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561047 2024-05-15T12:39:28.444 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:28.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561047 2024-05-15T12:39:28.445 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.8 seq 188978561047 2024-05-15T12:39:28.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561047' 2024-05-15T12:39:28.445 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-15T12:39:28.954 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561048 -lt 188978561047 2024-05-15T12:39:28.955 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:28.956 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-210453397522 2024-05-15T12:39:28.956 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:28.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-05-15T12:39:28.958 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:28.958 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-210453397522 2024-05-15T12:39:28.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397522 2024-05-15T12:39:28.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 210453397522' 2024-05-15T12:39:28.960 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.9 seq 210453397522 2024-05-15T12:39:28.960 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-05-15T12:39:29.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397523 -lt 210453397522 2024-05-15T12:39:29.437 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:29.437 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-231928233990 2024-05-15T12:39:29.437 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:29.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-05-15T12:39:29.439 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-231928233990 2024-05-15T12:39:29.440 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:29.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928233990 2024-05-15T12:39:29.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 231928233990' 2024-05-15T12:39:29.441 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.10 seq 231928233990 2024-05-15T12:39:29.441 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-05-15T12:39:29.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928233991 -lt 231928233990 2024-05-15T12:39:29.929 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T12:39:29.929 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:39:29.930 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:39:30.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-15T12:39:30.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T12:39:30.591 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T12:39:30.591 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T12:39:30.591 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T12:39:30.591 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T12:39:30.591 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T12:39:30.591 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T12:39:31.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-15T12:39:31.043 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T12:39:31.044 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:39:31.044 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:39:31.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-15T12:39:31.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T12:39:31.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T12:39:31.590 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: get_asok_path osd.0 2024-05-15T12:39:31.591 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T12:39:31.591 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T12:39:31.592 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:39:31.592 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:39:31.592 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:39:31.592 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48542/ceph-osd.0.asok 2024-05-15T12:39:31.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: CEPH_ARGS= 2024-05-15T12:39:31.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: ceph --admin-daemon /tmp/ceph-asok.48542/ceph-osd.0.asok log flush 2024-05-15T12:39:31.701 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:39:31.713 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:43: run: grep 'load: jerasure.*lrc' td/test-erasure-code/osd.0.log 2024-05-15T12:39:31.719 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:32:35.757+0000 7f4f1b28d7c0 0 load: jerasure load: lrc 2024-05-15T12:39:31.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:44: run: create_erasure_coded_pool ecpool 2024-05-15T12:39:31.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:56: create_erasure_coded_pool: local poolname=ecpool 2024-05-15T12:39:31.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:58: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd 2024-05-15T12:39:32.442 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:60: create_erasure_coded_pool: create_pool ecpool 12 12 erasure myprofile 2024-05-15T12:39:32.442 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 12 12 erasure myprofile 2024-05-15T12:39:33.255 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'ecpool' already exists 2024-05-15T12:39:33.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T12:39:34.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:62: create_erasure_coded_pool: wait_for_clean 2024-05-15T12:39:34.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T12:39:34.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T12:39:34.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T12:39:34.275 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T12:39:34.276 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T12:39:34.277 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T12:39:34.277 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T12:39:34.277 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T12:39:34.277 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T12:39:34.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T12:39:34.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T12:39:34.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T12:39:34.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T12:39:34.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T12:39:34.403 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T12:39:35.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T12:39:35.634 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:39:35.634 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T12:39:35.634 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:39:35.634 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T12:39:35.635 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T12:39:35.635 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T12:39:35.635 INFO:tasks.workunit.client.0.smithi005.stderr:7 2024-05-15T12:39:35.635 INFO:tasks.workunit.client.0.smithi005.stderr:8 2024-05-15T12:39:35.635 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:39:35.635 INFO:tasks.workunit.client.0.smithi005.stderr:10' 2024-05-15T12:39:35.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T12:39:35.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:35.635 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T12:39:35.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836565 2024-05-15T12:39:35.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836565 2024-05-15T12:39:35.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565' 2024-05-15T12:39:35.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:35.800 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T12:39:36.164 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673041 2024-05-15T12:39:36.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673041 2024-05-15T12:39:36.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673041' 2024-05-15T12:39:36.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:36.165 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T12:39:36.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509511 2024-05-15T12:39:36.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509511 2024-05-15T12:39:36.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673041 2-64424509511' 2024-05-15T12:39:36.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:36.541 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T12:39:37.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345984 2024-05-15T12:39:37.134 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345984 2024-05-15T12:39:37.134 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673041 2-64424509511 3-85899345984' 2024-05-15T12:39:37.134 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:37.134 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T12:39:37.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182458 2024-05-15T12:39:37.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182458 2024-05-15T12:39:37.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673041 2-64424509511 3-85899345984 4-107374182458' 2024-05-15T12:39:37.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:37.299 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-15T12:39:37.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051635 2024-05-15T12:39:37.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051635 2024-05-15T12:39:37.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673041 2-64424509511 3-85899345984 4-107374182458 5-124554051635' 2024-05-15T12:39:37.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:37.465 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-15T12:39:37.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888110 2024-05-15T12:39:37.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888110 2024-05-15T12:39:37.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673041 2-64424509511 3-85899345984 4-107374182458 5-124554051635 6-146028888110' 2024-05-15T12:39:37.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:37.721 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-15T12:39:37.910 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724582 2024-05-15T12:39:37.910 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724582 2024-05-15T12:39:37.910 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673041 2-64424509511 3-85899345984 4-107374182458 5-124554051635 6-146028888110 7-167503724582' 2024-05-15T12:39:37.910 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:37.911 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-15T12:39:38.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561052 2024-05-15T12:39:38.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561052 2024-05-15T12:39:38.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673041 2-64424509511 3-85899345984 4-107374182458 5-124554051635 6-146028888110 7-167503724582 8-188978561052' 2024-05-15T12:39:38.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:38.087 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-05-15T12:39:38.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397527 2024-05-15T12:39:38.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397527 2024-05-15T12:39:38.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673041 2-64424509511 3-85899345984 4-107374182458 5-124554051635 6-146028888110 7-167503724582 8-188978561052 9-210453397527' 2024-05-15T12:39:38.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:38.247 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-05-15T12:39:38.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928233995 2024-05-15T12:39:38.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928233995 2024-05-15T12:39:38.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673041 2-64424509511 3-85899345984 4-107374182458 5-124554051635 6-146028888110 7-167503724582 8-188978561052 9-210453397527 10-231928233995' 2024-05-15T12:39:38.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:38.443 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836565 2024-05-15T12:39:38.444 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:38.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T12:39:38.445 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836565 2024-05-15T12:39:38.445 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:38.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836565 2024-05-15T12:39:38.446 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 21474836565 2024-05-15T12:39:38.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836565' 2024-05-15T12:39:38.447 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T12:39:38.894 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836566 -lt 21474836565 2024-05-15T12:39:38.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:38.895 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673041 2024-05-15T12:39:38.896 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:38.897 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T12:39:38.898 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673041 2024-05-15T12:39:38.898 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:38.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673041 2024-05-15T12:39:38.899 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 42949673041 2024-05-15T12:39:38.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673041' 2024-05-15T12:39:38.899 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T12:39:39.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673041 -lt 42949673041 2024-05-15T12:39:39.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:39.344 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509511 2024-05-15T12:39:39.344 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:39.345 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T12:39:39.346 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509511 2024-05-15T12:39:39.346 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:39.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509511 2024-05-15T12:39:39.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509511' 2024-05-15T12:39:39.347 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 64424509511 2024-05-15T12:39:39.348 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T12:39:39.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509512 -lt 64424509511 2024-05-15T12:39:39.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:39.801 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345984 2024-05-15T12:39:39.801 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:39.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T12:39:39.804 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345984 2024-05-15T12:39:39.804 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:39.806 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345984 2024-05-15T12:39:39.806 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 85899345984 2024-05-15T12:39:39.806 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345984' 2024-05-15T12:39:39.806 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T12:39:40.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345985 -lt 85899345984 2024-05-15T12:39:40.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:40.242 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182458 2024-05-15T12:39:40.242 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:40.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T12:39:40.245 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182458 2024-05-15T12:39:40.245 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:40.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182458 2024-05-15T12:39:40.246 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.4 seq 107374182458 2024-05-15T12:39:40.247 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182458' 2024-05-15T12:39:40.247 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T12:39:40.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182458 -lt 107374182458 2024-05-15T12:39:40.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:40.750 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-124554051635 2024-05-15T12:39:40.750 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:40.752 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-15T12:39:40.753 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-124554051635 2024-05-15T12:39:40.754 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:40.755 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051635 2024-05-15T12:39:40.755 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.5 seq 124554051635 2024-05-15T12:39:40.755 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 124554051635' 2024-05-15T12:39:40.755 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-15T12:39:41.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051635 -lt 124554051635 2024-05-15T12:39:41.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:41.201 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888110 2024-05-15T12:39:41.201 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:41.202 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-15T12:39:41.203 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888110 2024-05-15T12:39:41.203 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:41.205 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888110 2024-05-15T12:39:41.205 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.6 seq 146028888110 2024-05-15T12:39:41.205 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888110' 2024-05-15T12:39:41.205 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-15T12:39:41.662 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888111 -lt 146028888110 2024-05-15T12:39:41.662 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:41.663 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724582 2024-05-15T12:39:41.663 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:41.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-05-15T12:39:41.665 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724582 2024-05-15T12:39:41.665 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:41.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724582 2024-05-15T12:39:41.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724582' 2024-05-15T12:39:41.667 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.7 seq 167503724582 2024-05-15T12:39:41.667 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-05-15T12:39:42.111 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724583 -lt 167503724582 2024-05-15T12:39:42.111 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:42.112 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561052 2024-05-15T12:39:42.112 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:42.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-15T12:39:42.115 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561052 2024-05-15T12:39:42.115 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:42.116 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.8 seq 188978561052 2024-05-15T12:39:42.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561052 2024-05-15T12:39:42.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561052' 2024-05-15T12:39:42.117 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-15T12:39:42.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561053 -lt 188978561052 2024-05-15T12:39:42.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:42.536 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-210453397527 2024-05-15T12:39:42.536 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:42.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-05-15T12:39:42.538 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-210453397527 2024-05-15T12:39:42.538 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:42.539 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397527 2024-05-15T12:39:42.539 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.9 seq 210453397527 2024-05-15T12:39:42.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 210453397527' 2024-05-15T12:39:42.540 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-05-15T12:39:42.987 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397528 -lt 210453397527 2024-05-15T12:39:42.988 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:42.988 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-231928233995 2024-05-15T12:39:42.988 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:42.989 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-05-15T12:39:42.990 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-231928233995 2024-05-15T12:39:42.990 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:42.991 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928233995 2024-05-15T12:39:42.991 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.10 seq 231928233995 2024-05-15T12:39:42.992 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 231928233995' 2024-05-15T12:39:42.992 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-05-15T12:39:43.415 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928233996 -lt 231928233995 2024-05-15T12:39:43.415 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T12:39:43.416 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:39:43.416 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:39:43.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-05-15T12:39:43.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T12:39:43.968 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T12:39:43.968 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T12:39:43.968 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T12:39:43.968 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T12:39:43.968 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T12:39:43.968 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T12:39:44.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-05-15T12:39:44.415 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T12:39:44.415 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:39:44.415 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:39:44.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-05-15T12:39:44.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T12:39:44.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T12:39:44.976 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: set 2024-05-15T12:39:44.976 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-05-15T12:39:44.978 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: local 'funcs=TEST_alignment_constraints 2024-05-15T12:39:44.978 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_chunk_mapping 2024-05-15T12:39:44.979 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_rados_put_get_isa 2024-05-15T12:39:44.979 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_rados_put_get_jerasure 2024-05-15T12:39:44.979 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_rados_put_get_lrc_advanced 2024-05-15T12:39:44.979 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_rados_put_get_lrc_kml 2024-05-15T12:39:44.979 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_rados_put_get_shec' 2024-05-15T12:39:44.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-05-15T12:39:44.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_alignment_constraints td/test-erasure-code 2024-05-15T12:39:44.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: TEST_alignment_constraints: local payload=ABC 2024-05-15T12:39:44.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:254: TEST_alignment_constraints: echo ABC 2024-05-15T12:39:44.979 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-05-15T12:39:45.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: local stripe_unit=4096 2024-05-15T12:39:45.001 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: ceph osd erasure-code-profile get myprofile 2024-05-15T12:39:45.001 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: grep k= 2024-05-15T12:39:45.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: eval local k=2 2024-05-15T12:39:45.429 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: local k=2 2024-05-15T12:39:45.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: TEST_alignment_constraints: local block_size=8191 2024-05-15T12:39:45.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:263: TEST_alignment_constraints: dd if=/dev/zero of=td/test-erasure-code/ORIGINAL bs=8191 count=2 2024-05-15T12:39:45.429 INFO:tasks.workunit.client.0.smithi005.stderr:2+0 records in 2024-05-15T12:39:45.429 INFO:tasks.workunit.client.0.smithi005.stderr:2+0 records out 2024-05-15T12:39:45.429 INFO:tasks.workunit.client.0.smithi005.stderr:16382 bytes (16 kB, 16 KiB) copied, 0.000169633 s, 96.6 MB/s 2024-05-15T12:39:45.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:264: TEST_alignment_constraints: rados --block-size=8191 --pool ecpool put UNALIGNED td/test-erasure-code/ORIGINAL 2024-05-15T12:39:45.471 INFO:tasks.workunit.client.0.smithi005.stderr:INFO: op_size has been rounded to 8192 2024-05-15T12:39:45.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: TEST_alignment_constraints: rm td/test-erasure-code/ORIGINAL 2024-05-15T12:39:45.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-05-15T12:39:45.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_chunk_mapping td/test-erasure-code 2024-05-15T12:39:45.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:304: TEST_chunk_mapping: local dir=td/test-erasure-code 2024-05-15T12:39:45.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:311: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code ecpool 0 1 2024-05-15T12:39:45.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:281: verify_chunk_mapping: local dir=td/test-erasure-code 2024-05-15T12:39:45.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local poolname=ecpool 2024-05-15T12:39:45.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: local first=0 2024-05-15T12:39:45.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:284: verify_chunk_mapping: local second=1 2024-05-15T12:39:45.926 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-05-15T12:39:45.926 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-05-15T12:39:45.949 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-05-15T12:39:45.949 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 FIRSTecpool 2024-05-15T12:39:45.950 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-05-15T12:39:45.950 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-05-15T12:39:45.969 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-05-15T12:39:45.969 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 SECONDecpool 2024-05-15T12:39:45.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: local 'payload= FIRSTecpool SECONDecpool' 2024-05-15T12:39:45.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: echo -n ' FIRSTecpool SECONDecpool' 2024-05-15T12:39:45.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: rados --pool ecpool put SOMETHINGecpool td/test-erasure-code/ORIGINAL 2024-05-15T12:39:46.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: rados --pool ecpool get SOMETHINGecpool td/test-erasure-code/COPY 2024-05-15T12:39:46.314 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2024-05-15T12:39:46.314 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-05-15T12:39:46.314 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGecpool 2024-05-15T12:39:46.315 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2024-05-15T12:39:46.315 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:39:46.744 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T12:39:46.744 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T12:39:46.744 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:39:46.745 INFO:tasks.workunit.client.0.smithi005.stderr:0' 2024-05-15T12:39:46.745 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 2024-05-15T12:39:46.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: osds=('3' '5' '9' '0') 2024-05-15T12:39:46.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local -a osds 2024-05-15T12:39:46.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i = 0 )) 2024-05-15T12:39:46.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-05-15T12:39:46.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.3 flush_journal 2024-05-15T12:39:46.873 INFO:tasks.workunit.client.0.smithi005.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-05-15T12:39:46.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-05-15T12:39:46.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-05-15T12:39:46.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.5 flush_journal 2024-05-15T12:39:47.008 INFO:tasks.workunit.client.0.smithi005.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-05-15T12:39:47.010 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-05-15T12:39:47.010 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-05-15T12:39:47.010 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2024-05-15T12:39:47.141 INFO:tasks.workunit.client.0.smithi005.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-05-15T12:39:47.142 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-05-15T12:39:47.142 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-05-15T12:39:47.143 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.0 flush_journal 2024-05-15T12:39:47.271 INFO:tasks.workunit.client.0.smithi005.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-05-15T12:39:47.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-05-15T12:39:47.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-05-15T12:39:47.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-15T12:39:47.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: rm td/test-erasure-code/COPY 2024-05-15T12:39:47.284 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2024-05-15T12:39:47.284 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-05-15T12:39:47.284 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGecpool 2024-05-15T12:39:47.284 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2024-05-15T12:39:47.284 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:39:47.703 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T12:39:47.703 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T12:39:47.703 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:39:47.704 INFO:tasks.workunit.client.0.smithi005.stderr:0' 2024-05-15T12:39:47.704 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 2024-05-15T12:39:47.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: osds=('3' '5' '9' '0') 2024-05-15T12:39:47.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: local -a osds 2024-05-15T12:39:47.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: grep --quiet FIRSTecpool 2024-05-15T12:39:47.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: objectstore_tool td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-05-15T12:39:47.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-05-15T12:39:47.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-15T12:39:47.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-05-15T12:39:47.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-15T12:39:47.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-05-15T12:39:47.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-05-15T12:39:47.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-15T12:39:47.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-05-15T12:39:47.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-15T12:39:47.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.3 2024-05-15T12:39:47.706 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:39:47.706 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:39:47.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:39:47.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:39:47.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:39:48.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:39:48.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-05-15T12:39:48.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-05-15T12:39:48.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-15T12:39:48.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-05-15T12:39:48.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-15T12:39:48.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/3 2024-05-15T12:39:48.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/3 SOMETHINGecpool get-bytes 2024-05-15T12:39:50.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 3 2024-05-15T12:39:50.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-05-15T12:39:50.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-15T12:39:50.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-05-15T12:39:50.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-15T12:39:50.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/3 2024-05-15T12:39:50.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-15T12:39:50.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:39:50.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:39:50.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:39:50.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2024-05-15T12:39:50.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2024-05-15T12:39:50.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-15T12:39:50.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-15T12:39:50.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-15T12:39:50.601 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-15T12:39:50.601 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:39:50.601 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:39:50.601 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:39:50.601 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:39:50.601 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:39:50.601 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:39:50.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:39:50.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:39:50.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-15T12:39:50.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-15T12:39:50.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:39:50.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:39:50.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:39:50.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:39:50.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-15T12:39:50.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-15T12:39:50.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/3 2024-05-15T12:39:50.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-05-15T12:39:50.603 INFO:tasks.workunit.client.0.smithi005.stderr:start osd.3 2024-05-15T12:39:50.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:39:50.603 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/3/whoami 2024-05-15T12:39:50.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-05-15T12:39:50.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-15T12:39:50.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-15T12:39:50.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-15T12:39:50.636 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:39:50.649+0000 7f6d796e17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:39:50.639 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:39:50.649+0000 7f6d796e17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:39:50.641 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:39:50.653+0000 7f6d796e17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:39:51.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-05-15T12:39:51.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:39:51.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-15T12:39:51.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:39:51.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:39:51.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:39:51.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:39:51.073 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T12:39:51.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:39:51.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:39:51.197 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:39:51.209+0000 7f6d796e17c0 -1 Falling back to public interface 2024-05-15T12:39:51.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:39:52.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:39:52.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:39:52.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:39:52.522 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:39:52.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:39:52.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:39:52.972 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:39:53.858 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:39:53.869+0000 7f6d796e17c0 -1 osd.3 74 log_to_monitors true 2024-05-15T12:39:53.974 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:39:53.974 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:39:53.974 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:39:53.974 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T12:39:53.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:39:53.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:39:54.432 INFO:tasks.workunit.client.0.smithi005.stderr:osd.3 up in weight 1 up_from 78 up_thru 67 down_at 75 last_clean_interval [20,74) [v2:127.0.0.1:6826/3922340757,v1:127.0.0.1:6827/3922340757] [v2:127.0.0.1:6828/3922340757,v1:127.0.0.1:6829/3922340757] exists,up 26cfd0e8-10b6-4c44-988f-992a5fea5779 2024-05-15T12:39:54.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:39:54.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:39:54.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:39:54.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-15T12:39:54.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T12:39:54.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T12:39:54.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T12:39:54.432 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T12:39:54.433 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T12:39:54.433 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T12:39:54.433 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T12:39:54.433 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T12:39:54.433 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T12:39:54.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T12:39:54.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T12:39:54.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T12:39:54.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T12:39:54.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T12:39:54.538 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T12:39:55.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T12:39:55.309 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:39:55.310 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T12:39:55.310 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:39:55.310 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T12:39:55.310 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T12:39:55.310 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T12:39:55.310 INFO:tasks.workunit.client.0.smithi005.stderr:7 2024-05-15T12:39:55.310 INFO:tasks.workunit.client.0.smithi005.stderr:8 2024-05-15T12:39:55.310 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:39:55.310 INFO:tasks.workunit.client.0.smithi005.stderr:10' 2024-05-15T12:39:55.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T12:39:55.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:55.311 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T12:39:55.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836571 2024-05-15T12:39:55.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836571 2024-05-15T12:39:55.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571' 2024-05-15T12:39:55.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:55.625 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T12:39:55.889 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673047 2024-05-15T12:39:55.889 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673047 2024-05-15T12:39:55.889 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673047' 2024-05-15T12:39:55.889 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:55.889 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T12:39:56.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509517 2024-05-15T12:39:56.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509517 2024-05-15T12:39:56.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673047 2-64424509517' 2024-05-15T12:39:56.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:56.189 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T12:39:56.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449091 2024-05-15T12:39:56.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449091 2024-05-15T12:39:56.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673047 2-64424509517 3-335007449091' 2024-05-15T12:39:56.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:56.350 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T12:39:56.512 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182464 2024-05-15T12:39:56.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182464 2024-05-15T12:39:56.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673047 2-64424509517 3-335007449091 4-107374182464' 2024-05-15T12:39:56.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:56.513 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-15T12:39:56.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051641 2024-05-15T12:39:56.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051641 2024-05-15T12:39:56.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673047 2-64424509517 3-335007449091 4-107374182464 5-124554051641' 2024-05-15T12:39:56.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:56.668 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-15T12:39:56.825 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888116 2024-05-15T12:39:56.825 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888116 2024-05-15T12:39:56.825 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673047 2-64424509517 3-335007449091 4-107374182464 5-124554051641 6-146028888116' 2024-05-15T12:39:56.825 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:56.826 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-15T12:39:57.028 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724588 2024-05-15T12:39:57.028 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724588 2024-05-15T12:39:57.028 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673047 2-64424509517 3-335007449091 4-107374182464 5-124554051641 6-146028888116 7-167503724588' 2024-05-15T12:39:57.028 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:57.028 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-15T12:39:57.211 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561058 2024-05-15T12:39:57.211 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561058 2024-05-15T12:39:57.211 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673047 2-64424509517 3-335007449091 4-107374182464 5-124554051641 6-146028888116 7-167503724588 8-188978561058' 2024-05-15T12:39:57.211 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:57.211 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-05-15T12:39:57.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397533 2024-05-15T12:39:57.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397533 2024-05-15T12:39:57.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673047 2-64424509517 3-335007449091 4-107374182464 5-124554051641 6-146028888116 7-167503724588 8-188978561058 9-210453397533' 2024-05-15T12:39:57.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:39:57.365 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-05-15T12:39:57.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928234001 2024-05-15T12:39:57.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928234001 2024-05-15T12:39:57.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673047 2-64424509517 3-335007449091 4-107374182464 5-124554051641 6-146028888116 7-167503724588 8-188978561058 9-210453397533 10-231928234001' 2024-05-15T12:39:57.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:57.521 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836571 2024-05-15T12:39:57.522 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:57.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T12:39:57.524 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836571 2024-05-15T12:39:57.524 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:57.525 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836571 2024-05-15T12:39:57.525 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836571' 2024-05-15T12:39:57.525 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.0 seq 21474836571 2024-05-15T12:39:57.525 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T12:39:57.946 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836571 -lt 21474836571 2024-05-15T12:39:57.946 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:57.947 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:57.947 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673047 2024-05-15T12:39:57.948 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T12:39:57.949 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673047 2024-05-15T12:39:57.949 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:57.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673047 2024-05-15T12:39:57.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673047' 2024-05-15T12:39:57.950 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.1 seq 42949673047 2024-05-15T12:39:57.951 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T12:39:58.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673047 -lt 42949673047 2024-05-15T12:39:58.389 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:58.389 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:58.389 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509517 2024-05-15T12:39:58.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T12:39:58.391 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509517 2024-05-15T12:39:58.391 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:58.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509517 2024-05-15T12:39:58.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509517' 2024-05-15T12:39:58.393 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.2 seq 64424509517 2024-05-15T12:39:58.393 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T12:39:58.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509518 -lt 64424509517 2024-05-15T12:39:58.845 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:58.845 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-335007449091 2024-05-15T12:39:58.845 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:58.847 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T12:39:58.847 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-335007449091 2024-05-15T12:39:58.848 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:58.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449091 2024-05-15T12:39:58.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 335007449091' 2024-05-15T12:39:58.850 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.3 seq 335007449091 2024-05-15T12:39:58.850 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T12:39:59.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449091 -lt 335007449091 2024-05-15T12:39:59.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:59.265 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182464 2024-05-15T12:39:59.265 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:59.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T12:39:59.268 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182464 2024-05-15T12:39:59.268 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:59.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182464 2024-05-15T12:39:59.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182464' 2024-05-15T12:39:59.269 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.4 seq 107374182464 2024-05-15T12:39:59.269 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T12:39:59.688 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182464 -lt 107374182464 2024-05-15T12:39:59.688 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:39:59.689 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-124554051641 2024-05-15T12:39:59.689 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:39:59.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-15T12:39:59.691 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-124554051641 2024-05-15T12:39:59.691 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:39:59.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051641 2024-05-15T12:39:59.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 124554051641' 2024-05-15T12:39:59.692 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.5 seq 124554051641 2024-05-15T12:39:59.693 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-15T12:40:00.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051641 -lt 124554051641 2024-05-15T12:40:00.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:00.115 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888116 2024-05-15T12:40:00.115 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:00.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-15T12:40:00.117 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888116 2024-05-15T12:40:00.117 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:00.119 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888116 2024-05-15T12:40:00.119 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888116' 2024-05-15T12:40:00.119 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.6 seq 146028888116 2024-05-15T12:40:00.119 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-15T12:40:00.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888116 -lt 146028888116 2024-05-15T12:40:00.528 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:00.528 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724588 2024-05-15T12:40:00.528 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:00.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-05-15T12:40:00.531 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724588 2024-05-15T12:40:00.531 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:00.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724588 2024-05-15T12:40:00.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724588' 2024-05-15T12:40:00.533 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.7 seq 167503724588 2024-05-15T12:40:00.533 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-05-15T12:40:00.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724589 -lt 167503724588 2024-05-15T12:40:00.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:00.957 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561058 2024-05-15T12:40:00.957 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:00.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-15T12:40:00.960 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561058 2024-05-15T12:40:00.960 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:00.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561058 2024-05-15T12:40:00.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561058' 2024-05-15T12:40:00.962 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.8 seq 188978561058 2024-05-15T12:40:00.962 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-15T12:40:01.406 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561059 -lt 188978561058 2024-05-15T12:40:01.407 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:01.407 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-210453397533 2024-05-15T12:40:01.407 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:01.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-05-15T12:40:01.409 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-210453397533 2024-05-15T12:40:01.409 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:01.410 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397533 2024-05-15T12:40:01.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 210453397533' 2024-05-15T12:40:01.411 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.9 seq 210453397533 2024-05-15T12:40:01.411 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-05-15T12:40:01.848 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397533 -lt 210453397533 2024-05-15T12:40:01.848 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:01.849 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-231928234001 2024-05-15T12:40:01.849 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:01.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-05-15T12:40:01.852 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-231928234001 2024-05-15T12:40:01.852 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:01.853 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928234001 2024-05-15T12:40:01.853 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 231928234001' 2024-05-15T12:40:01.853 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.10 seq 231928234001 2024-05-15T12:40:01.854 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-05-15T12:40:02.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928234001 -lt 231928234001 2024-05-15T12:40:02.283 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T12:40:02.283 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:40:02.283 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:40:02.840 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-05-15T12:40:02.841 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T12:40:02.841 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T12:40:02.842 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T12:40:02.842 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T12:40:02.842 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T12:40:02.842 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T12:40:02.842 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T12:40:03.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-05-15T12:40:03.273 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T12:40:03.273 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:40:03.273 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:40:03.810 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-05-15T12:40:03.811 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T12:40:03.811 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T12:40:03.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: objectstore_tool td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-05-15T12:40:03.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-05-15T12:40:03.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-15T12:40:03.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-05-15T12:40:03.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-15T12:40:03.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-05-15T12:40:03.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-05-15T12:40:03.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-15T12:40:03.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-05-15T12:40:03.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-15T12:40:03.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.5 2024-05-15T12:40:03.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: grep --quiet SECONDecpool 2024-05-15T12:40:03.814 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:40:03.814 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:40:03.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:40:03.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:40:03.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:40:04.122 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:40:04.122 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-05-15T12:40:04.122 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-05-15T12:40:04.122 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-15T12:40:04.122 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-05-15T12:40:04.122 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-15T12:40:04.122 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/5 2024-05-15T12:40:04.122 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/5 SOMETHINGecpool get-bytes 2024-05-15T12:40:07.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 5 2024-05-15T12:40:07.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-05-15T12:40:07.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-15T12:40:07.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-05-15T12:40:07.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-15T12:40:07.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/5 2024-05-15T12:40:07.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-15T12:40:07.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:40:07.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:40:07.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:40:07.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2024-05-15T12:40:07.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2024-05-15T12:40:07.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-15T12:40:07.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-15T12:40:07.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-15T12:40:07.505 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-15T12:40:07.505 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:40:07.505 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:40:07.505 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:40:07.505 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:40:07.505 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:40:07.505 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:40:07.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:40:07.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:40:07.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-15T12:40:07.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-15T12:40:07.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:40:07.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:40:07.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:40:07.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:40:07.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-15T12:40:07.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-15T12:40:07.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/5 2024-05-15T12:40:07.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-05-15T12:40:07.507 INFO:tasks.workunit.client.0.smithi005.stderr:start osd.5 2024-05-15T12:40:07.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:40:07.508 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/5/whoami 2024-05-15T12:40:07.508 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-05-15T12:40:07.508 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-15T12:40:07.508 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-15T12:40:07.509 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-15T12:40:07.542 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:40:07.553+0000 7fab147057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:40:07.545 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:40:07.557+0000 7fab147057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:40:07.547 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:40:07.557+0000 7fab147057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:40:07.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-05-15T12:40:07.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:40:07.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-05-15T12:40:07.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:40:07.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:40:07.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:40:07.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:40:07.939 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T12:40:07.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:40:07.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:40:08.117 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:40:08.129+0000 7fab147057c0 -1 Falling back to public interface 2024-05-15T12:40:08.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:40:09.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:40:09.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:40:09.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:40:09.365 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:40:09.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:40:09.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:40:09.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:40:10.662 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:40:10.673+0000 7fab147057c0 -1 osd.5 80 log_to_monitors true 2024-05-15T12:40:10.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:40:10.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:40:10.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:40:10.796 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T12:40:10.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:40:10.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:40:11.425 INFO:tasks.workunit.client.0.smithi005.stderr:osd.5 up in weight 1 up_from 85 up_thru 78 down_at 81 last_clean_interval [29,80) [v2:127.0.0.1:6842/2260430933,v1:127.0.0.1:6843/2260430933] [v2:127.0.0.1:6844/2260430933,v1:127.0.0.1:6845/2260430933] exists,up 5330abbc-8264-44a7-a2a3-af755956921b 2024-05-15T12:40:11.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:40:11.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:40:11.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:40:11.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-15T12:40:11.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T12:40:11.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T12:40:11.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T12:40:11.426 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T12:40:11.426 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T12:40:11.426 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T12:40:11.427 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T12:40:11.427 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T12:40:11.427 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T12:40:11.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T12:40:11.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T12:40:11.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T12:40:11.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T12:40:11.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T12:40:11.540 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T12:40:12.716 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T12:40:12.716 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:40:12.716 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T12:40:12.716 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:40:12.717 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T12:40:12.717 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T12:40:12.717 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T12:40:12.717 INFO:tasks.workunit.client.0.smithi005.stderr:7 2024-05-15T12:40:12.717 INFO:tasks.workunit.client.0.smithi005.stderr:8 2024-05-15T12:40:12.717 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:40:12.717 INFO:tasks.workunit.client.0.smithi005.stderr:10' 2024-05-15T12:40:12.717 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T12:40:12.717 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:12.717 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T12:40:12.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836576 2024-05-15T12:40:12.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836576 2024-05-15T12:40:12.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576' 2024-05-15T12:40:12.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:12.882 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T12:40:13.136 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673052 2024-05-15T12:40:13.136 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673052 2024-05-15T12:40:13.136 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673052' 2024-05-15T12:40:13.136 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:13.136 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T12:40:13.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509523 2024-05-15T12:40:13.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509523 2024-05-15T12:40:13.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673052 2-64424509523' 2024-05-15T12:40:13.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:13.631 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T12:40:13.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449096 2024-05-15T12:40:13.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449096 2024-05-15T12:40:13.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673052 2-64424509523 3-335007449096' 2024-05-15T12:40:13.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:13.917 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T12:40:14.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182469 2024-05-15T12:40:14.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182469 2024-05-15T12:40:14.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673052 2-64424509523 3-335007449096 4-107374182469' 2024-05-15T12:40:14.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:14.244 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-15T12:40:14.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220163 2024-05-15T12:40:14.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220163 2024-05-15T12:40:14.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673052 2-64424509523 3-335007449096 4-107374182469 5-365072220163' 2024-05-15T12:40:14.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:14.402 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-15T12:40:14.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888122 2024-05-15T12:40:14.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888122 2024-05-15T12:40:14.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673052 2-64424509523 3-335007449096 4-107374182469 5-365072220163 6-146028888122' 2024-05-15T12:40:14.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:14.587 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-15T12:40:14.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724594 2024-05-15T12:40:14.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724594 2024-05-15T12:40:14.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673052 2-64424509523 3-335007449096 4-107374182469 5-365072220163 6-146028888122 7-167503724594' 2024-05-15T12:40:14.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:14.830 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-15T12:40:15.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561064 2024-05-15T12:40:15.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561064 2024-05-15T12:40:15.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673052 2-64424509523 3-335007449096 4-107374182469 5-365072220163 6-146028888122 7-167503724594 8-188978561064' 2024-05-15T12:40:15.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:15.015 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-05-15T12:40:15.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397538 2024-05-15T12:40:15.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397538 2024-05-15T12:40:15.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673052 2-64424509523 3-335007449096 4-107374182469 5-365072220163 6-146028888122 7-167503724594 8-188978561064 9-210453397538' 2024-05-15T12:40:15.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:15.199 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-05-15T12:40:15.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928234006 2024-05-15T12:40:15.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928234006 2024-05-15T12:40:15.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673052 2-64424509523 3-335007449096 4-107374182469 5-365072220163 6-146028888122 7-167503724594 8-188978561064 9-210453397538 10-231928234006' 2024-05-15T12:40:15.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:15.354 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836576 2024-05-15T12:40:15.355 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:15.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T12:40:15.355 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836576 2024-05-15T12:40:15.356 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:15.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836576 2024-05-15T12:40:15.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836576' 2024-05-15T12:40:15.357 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.0 seq 21474836576 2024-05-15T12:40:15.357 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T12:40:15.777 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836577 -lt 21474836576 2024-05-15T12:40:15.777 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:15.778 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673052 2024-05-15T12:40:15.778 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:15.782 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T12:40:15.782 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673052 2024-05-15T12:40:15.782 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:15.783 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673052 2024-05-15T12:40:15.783 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2024-05-15T12:40:15.783 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.1 seq 42949673052 2024-05-15T12:40:15.783 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T12:40:16.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673052 -lt 42949673052 2024-05-15T12:40:16.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:16.215 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509523 2024-05-15T12:40:16.216 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:16.216 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T12:40:16.217 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509523 2024-05-15T12:40:16.217 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:16.219 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509523 2024-05-15T12:40:16.219 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509523' 2024-05-15T12:40:16.219 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.2 seq 64424509523 2024-05-15T12:40:16.219 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T12:40:16.650 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509523 -lt 64424509523 2024-05-15T12:40:16.650 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:16.651 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-335007449096 2024-05-15T12:40:16.651 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:16.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T12:40:16.653 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-335007449096 2024-05-15T12:40:16.653 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:16.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449096 2024-05-15T12:40:16.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 335007449096' 2024-05-15T12:40:16.655 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.3 seq 335007449096 2024-05-15T12:40:16.655 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T12:40:17.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449097 -lt 335007449096 2024-05-15T12:40:17.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:17.086 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182469 2024-05-15T12:40:17.086 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:17.088 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T12:40:17.089 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182469 2024-05-15T12:40:17.089 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:17.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182469 2024-05-15T12:40:17.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182469' 2024-05-15T12:40:17.090 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.4 seq 107374182469 2024-05-15T12:40:17.090 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T12:40:17.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182470 -lt 107374182469 2024-05-15T12:40:17.514 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:17.514 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-365072220163 2024-05-15T12:40:17.514 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:17.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-15T12:40:17.516 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-365072220163 2024-05-15T12:40:17.516 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:17.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220163 2024-05-15T12:40:17.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 365072220163' 2024-05-15T12:40:17.518 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.5 seq 365072220163 2024-05-15T12:40:17.518 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-15T12:40:17.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220164 -lt 365072220163 2024-05-15T12:40:17.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:17.938 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888122 2024-05-15T12:40:17.938 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:17.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-15T12:40:17.940 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888122 2024-05-15T12:40:17.940 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:17.941 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888122 2024-05-15T12:40:17.942 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888122' 2024-05-15T12:40:17.942 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.6 seq 146028888122 2024-05-15T12:40:17.942 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-15T12:40:18.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888122 -lt 146028888122 2024-05-15T12:40:18.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:18.400 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724594 2024-05-15T12:40:18.400 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:18.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-05-15T12:40:18.403 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724594 2024-05-15T12:40:18.403 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:18.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724594 2024-05-15T12:40:18.404 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724594' 2024-05-15T12:40:18.404 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.7 seq 167503724594 2024-05-15T12:40:18.404 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-05-15T12:40:18.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724595 -lt 167503724594 2024-05-15T12:40:18.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:18.838 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:18.838 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561064 2024-05-15T12:40:18.839 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-15T12:40:18.839 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561064 2024-05-15T12:40:18.840 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:18.841 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561064 2024-05-15T12:40:18.841 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561064' 2024-05-15T12:40:18.841 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.8 seq 188978561064 2024-05-15T12:40:18.841 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-15T12:40:19.255 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561064 -lt 188978561064 2024-05-15T12:40:19.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:19.256 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:19.256 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-210453397538 2024-05-15T12:40:19.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-05-15T12:40:19.258 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-210453397538 2024-05-15T12:40:19.258 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:19.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397538 2024-05-15T12:40:19.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 210453397538' 2024-05-15T12:40:19.260 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.9 seq 210453397538 2024-05-15T12:40:19.260 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-05-15T12:40:19.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397539 -lt 210453397538 2024-05-15T12:40:19.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:19.677 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-231928234006 2024-05-15T12:40:19.677 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:19.678 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-05-15T12:40:19.679 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:19.679 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-231928234006 2024-05-15T12:40:19.680 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928234006 2024-05-15T12:40:19.680 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 231928234006' 2024-05-15T12:40:19.680 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.10 seq 231928234006 2024-05-15T12:40:19.680 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-05-15T12:40:20.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928234007 -lt 231928234006 2024-05-15T12:40:20.118 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T12:40:20.118 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:40:20.118 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:40:20.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-05-15T12:40:20.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T12:40:20.695 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T12:40:20.695 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T12:40:20.695 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T12:40:20.695 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T12:40:20.695 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T12:40:20.695 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T12:40:21.121 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-05-15T12:40:21.121 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T12:40:21.121 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:40:21.122 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:40:21.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-05-15T12:40:21.656 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T12:40:21.656 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T12:40:21.656 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:313: TEST_chunk_mapping: ceph osd erasure-code-profile set remap-profile plugin=lrc 'layers=[ [ "cDD", "" ] ]' mapping=_DD 'crush-steps=[ [ "choose", "osd", 0 ] ]' 2024-05-15T12:40:22.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:318: TEST_chunk_mapping: ceph osd erasure-code-profile get remap-profile 2024-05-15T12:40:22.689 INFO:tasks.workunit.client.0.smithi005.stdout:crush-device-class= 2024-05-15T12:40:22.689 INFO:tasks.workunit.client.0.smithi005.stdout:crush-failure-domain=host 2024-05-15T12:40:22.689 INFO:tasks.workunit.client.0.smithi005.stdout:crush-num-failure-domains=0 2024-05-15T12:40:22.689 INFO:tasks.workunit.client.0.smithi005.stdout:crush-osds-per-failure-domain=0 2024-05-15T12:40:22.689 INFO:tasks.workunit.client.0.smithi005.stdout:crush-root=default 2024-05-15T12:40:22.689 INFO:tasks.workunit.client.0.smithi005.stdout:crush-steps=[ [ "choose", "osd", 0 ] ] 2024-05-15T12:40:22.690 INFO:tasks.workunit.client.0.smithi005.stdout:k=-1 2024-05-15T12:40:22.690 INFO:tasks.workunit.client.0.smithi005.stdout:l=-1 2024-05-15T12:40:22.690 INFO:tasks.workunit.client.0.smithi005.stdout:layers=[ [ "cDD", "" ] ] 2024-05-15T12:40:22.690 INFO:tasks.workunit.client.0.smithi005.stdout:m=-1 2024-05-15T12:40:22.690 INFO:tasks.workunit.client.0.smithi005.stdout:mapping=_DD 2024-05-15T12:40:22.690 INFO:tasks.workunit.client.0.smithi005.stdout:plugin=lrc 2024-05-15T12:40:22.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:319: TEST_chunk_mapping: create_pool remap-pool 12 12 erasure remap-profile 2024-05-15T12:40:22.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create remap-pool 12 12 erasure remap-profile 2024-05-15T12:40:23.560 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'remap-pool' already exists 2024-05-15T12:40:23.578 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T12:40:24.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:327: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code remap-pool 1 2 2024-05-15T12:40:24.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:281: verify_chunk_mapping: local dir=td/test-erasure-code 2024-05-15T12:40:24.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local poolname=remap-pool 2024-05-15T12:40:24.580 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: local first=1 2024-05-15T12:40:24.580 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:284: verify_chunk_mapping: local second=2 2024-05-15T12:40:24.581 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-05-15T12:40:24.581 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-05-15T12:40:24.606 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-05-15T12:40:24.606 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 FIRSTremap-pool 2024-05-15T12:40:24.607 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-05-15T12:40:24.607 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-05-15T12:40:24.630 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-05-15T12:40:24.631 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 SECONDremap-pool 2024-05-15T12:40:24.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: local 'payload= FIRSTremap-pool SECONDremap-pool' 2024-05-15T12:40:24.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: echo -n ' FIRSTremap-pool SECONDremap-pool' 2024-05-15T12:40:24.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: rados --pool remap-pool put SOMETHINGremap-pool td/test-erasure-code/ORIGINAL 2024-05-15T12:40:26.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: rados --pool remap-pool get SOMETHINGremap-pool td/test-erasure-code/COPY 2024-05-15T12:40:27.126 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2024-05-15T12:40:27.126 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=remap-pool 2024-05-15T12:40:27.126 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGremap-pool 2024-05-15T12:40:27.127 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:40:27.127 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2024-05-15T12:40:27.692 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-05-15T12:40:27.692 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T12:40:27.692 INFO:tasks.workunit.client.0.smithi005.stderr:9' 2024-05-15T12:40:27.692 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 6 9 2024-05-15T12:40:27.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: osds=('4' '6' '9') 2024-05-15T12:40:27.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local -a osds 2024-05-15T12:40:27.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i = 0 )) 2024-05-15T12:40:27.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-05-15T12:40:27.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.4 flush_journal 2024-05-15T12:40:27.828 INFO:tasks.workunit.client.0.smithi005.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-05-15T12:40:27.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-05-15T12:40:27.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-05-15T12:40:27.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.6 flush_journal 2024-05-15T12:40:27.965 INFO:tasks.workunit.client.0.smithi005.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-05-15T12:40:27.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-05-15T12:40:27.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-05-15T12:40:27.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2024-05-15T12:40:28.101 INFO:tasks.workunit.client.0.smithi005.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-05-15T12:40:28.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-05-15T12:40:28.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-05-15T12:40:28.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-15T12:40:28.103 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: rm td/test-erasure-code/COPY 2024-05-15T12:40:28.105 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2024-05-15T12:40:28.106 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=remap-pool 2024-05-15T12:40:28.106 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGremap-pool 2024-05-15T12:40:28.106 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2024-05-15T12:40:28.106 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:40:28.726 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-05-15T12:40:28.727 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T12:40:28.727 INFO:tasks.workunit.client.0.smithi005.stderr:9' 2024-05-15T12:40:28.727 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 6 9 2024-05-15T12:40:28.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: osds=('4' '6' '9') 2024-05-15T12:40:28.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: local -a osds 2024-05-15T12:40:28.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: objectstore_tool td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-05-15T12:40:28.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-05-15T12:40:28.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-15T12:40:28.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=6 2024-05-15T12:40:28.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-15T12:40:28.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-05-15T12:40:28.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-05-15T12:40:28.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-15T12:40:28.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=6 2024-05-15T12:40:28.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-15T12:40:28.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.6 2024-05-15T12:40:28.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: grep --quiet FIRSTremap-pool 2024-05-15T12:40:28.729 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:40:28.729 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:40:28.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:40:28.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:40:28.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:40:30.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:40:30.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-05-15T12:40:30.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-05-15T12:40:30.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-15T12:40:30.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=6 2024-05-15T12:40:30.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-15T12:40:30.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/6 2024-05-15T12:40:30.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/6 SOMETHINGremap-pool get-bytes 2024-05-15T12:40:32.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 6 2024-05-15T12:40:32.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-05-15T12:40:32.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-15T12:40:32.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=6 2024-05-15T12:40:32.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-15T12:40:32.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/6 2024-05-15T12:40:32.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-15T12:40:32.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:40:32.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:40:32.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:40:32.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2024-05-15T12:40:32.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2024-05-15T12:40:32.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-15T12:40:32.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-15T12:40:32.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-15T12:40:32.660 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-15T12:40:32.660 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:40:32.660 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:40:32.660 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:40:32.660 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:40:32.660 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:40:32.660 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:40:32.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:40:32.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:40:32.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-15T12:40:32.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-15T12:40:32.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:40:32.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:40:32.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:40:32.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:40:32.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-15T12:40:32.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-15T12:40:32.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/6 2024-05-15T12:40:32.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.6 2024-05-15T12:40:32.662 INFO:tasks.workunit.client.0.smithi005.stderr:start osd.6 2024-05-15T12:40:32.662 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 6 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:40:32.662 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/6/whoami 2024-05-15T12:40:32.662 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 6 = 6 ']' 2024-05-15T12:40:32.662 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-15T12:40:32.662 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-15T12:40:32.662 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-15T12:40:32.695 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:40:32.704+0000 7fa618a287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:40:32.730 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:40:32.740+0000 7fa618a287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:40:32.732 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:40:32.744+0000 7fa618a287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:40:33.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 6 2024-05-15T12:40:33.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:40:33.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-05-15T12:40:33.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:40:33.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:40:33.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:40:33.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:40:33.108 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T12:40:33.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:40:33.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:40:33.306 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:40:33.316+0000 7fa618a287c0 -1 Falling back to public interface 2024-05-15T12:40:33.583 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:40:34.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:40:34.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:40:34.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:40:34.585 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:40:34.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:40:34.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:40:35.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:40:35.878 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:40:35.888+0000 7fa618a287c0 -1 osd.6 98 log_to_monitors true 2024-05-15T12:40:36.091 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:40:36.092 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:40:36.092 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:40:36.092 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T12:40:36.092 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:40:36.092 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:40:36.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:40:37.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:40:37.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:40:37.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:40:37.586 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:40:37.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:40:37.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:40:38.578 INFO:tasks.workunit.client.0.smithi005.stderr:osd.6 up in weight 1 up_from 102 up_thru 102 down_at 99 last_clean_interval [34,98) [v2:127.0.0.1:6850/3287763503,v1:127.0.0.1:6851/3287763503] [v2:127.0.0.1:6852/3287763503,v1:127.0.0.1:6853/3287763503] exists,up 7e2b0f2f-49ac-4c6a-860f-a361e974bd79 2024-05-15T12:40:38.578 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:40:38.578 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:40:38.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:40:38.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-15T12:40:38.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T12:40:38.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T12:40:38.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T12:40:38.579 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T12:40:38.580 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T12:40:38.580 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T12:40:38.580 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T12:40:38.580 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T12:40:38.580 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T12:40:38.687 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T12:40:38.687 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T12:40:38.687 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T12:40:38.687 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T12:40:38.687 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T12:40:38.687 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T12:40:39.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T12:40:39.488 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:40:39.488 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T12:40:39.488 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:40:39.489 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T12:40:39.489 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T12:40:39.489 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T12:40:39.489 INFO:tasks.workunit.client.0.smithi005.stderr:7 2024-05-15T12:40:39.489 INFO:tasks.workunit.client.0.smithi005.stderr:8 2024-05-15T12:40:39.489 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:40:39.489 INFO:tasks.workunit.client.0.smithi005.stderr:10' 2024-05-15T12:40:39.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T12:40:39.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:39.490 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T12:40:39.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836584 2024-05-15T12:40:39.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836584 2024-05-15T12:40:39.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584' 2024-05-15T12:40:39.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:39.756 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T12:40:40.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673059 2024-05-15T12:40:40.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673059 2024-05-15T12:40:40.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059' 2024-05-15T12:40:40.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:40.088 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T12:40:40.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509530 2024-05-15T12:40:40.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509530 2024-05-15T12:40:40.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509530' 2024-05-15T12:40:40.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:40.252 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T12:40:40.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449104 2024-05-15T12:40:40.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449104 2024-05-15T12:40:40.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509530 3-335007449104' 2024-05-15T12:40:40.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:40.409 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T12:40:40.561 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182477 2024-05-15T12:40:40.561 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182477 2024-05-15T12:40:40.561 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509530 3-335007449104 4-107374182477' 2024-05-15T12:40:40.562 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:40.562 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-15T12:40:40.715 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220170 2024-05-15T12:40:40.715 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220170 2024-05-15T12:40:40.715 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509530 3-335007449104 4-107374182477 5-365072220170' 2024-05-15T12:40:40.715 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:40.715 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-15T12:40:40.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664195 2024-05-15T12:40:40.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664195 2024-05-15T12:40:40.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509530 3-335007449104 4-107374182477 5-365072220170 6-438086664195' 2024-05-15T12:40:40.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:40.874 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-15T12:40:41.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724601 2024-05-15T12:40:41.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724601 2024-05-15T12:40:41.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509530 3-335007449104 4-107374182477 5-365072220170 6-438086664195 7-167503724601' 2024-05-15T12:40:41.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:41.079 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-15T12:40:41.230 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561071 2024-05-15T12:40:41.231 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561071 2024-05-15T12:40:41.231 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509530 3-335007449104 4-107374182477 5-365072220170 6-438086664195 7-167503724601 8-188978561071' 2024-05-15T12:40:41.231 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:41.231 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-05-15T12:40:41.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397545 2024-05-15T12:40:41.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397545 2024-05-15T12:40:41.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509530 3-335007449104 4-107374182477 5-365072220170 6-438086664195 7-167503724601 8-188978561071 9-210453397545' 2024-05-15T12:40:41.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:41.387 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-05-15T12:40:41.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928234014 2024-05-15T12:40:41.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928234014 2024-05-15T12:40:41.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509530 3-335007449104 4-107374182477 5-365072220170 6-438086664195 7-167503724601 8-188978561071 9-210453397545 10-231928234014' 2024-05-15T12:40:41.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:41.546 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836584 2024-05-15T12:40:41.546 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:41.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T12:40:41.547 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836584 2024-05-15T12:40:41.547 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:41.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836584 2024-05-15T12:40:41.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836584' 2024-05-15T12:40:41.548 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.0 seq 21474836584 2024-05-15T12:40:41.548 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T12:40:41.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836584 -lt 21474836584 2024-05-15T12:40:41.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:41.971 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673059 2024-05-15T12:40:41.971 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:41.972 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T12:40:41.973 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673059 2024-05-15T12:40:41.973 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:41.974 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673059 2024-05-15T12:40:41.974 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2024-05-15T12:40:41.974 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.1 seq 42949673059 2024-05-15T12:40:41.975 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T12:40:42.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673060 -lt 42949673059 2024-05-15T12:40:42.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:42.436 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509530 2024-05-15T12:40:42.436 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:42.437 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T12:40:42.438 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509530 2024-05-15T12:40:42.438 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:42.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509530 2024-05-15T12:40:42.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-05-15T12:40:42.440 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.2 seq 64424509530 2024-05-15T12:40:42.440 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T12:40:42.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509531 -lt 64424509530 2024-05-15T12:40:42.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:42.918 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-335007449104 2024-05-15T12:40:42.918 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:42.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T12:40:42.920 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-335007449104 2024-05-15T12:40:42.920 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:42.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449104 2024-05-15T12:40:42.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 335007449104' 2024-05-15T12:40:42.921 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.3 seq 335007449104 2024-05-15T12:40:42.921 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T12:40:43.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449104 -lt 335007449104 2024-05-15T12:40:43.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:43.382 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182477 2024-05-15T12:40:43.382 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:43.383 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T12:40:43.384 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182477 2024-05-15T12:40:43.384 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:43.385 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182477 2024-05-15T12:40:43.385 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182477' 2024-05-15T12:40:43.385 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.4 seq 107374182477 2024-05-15T12:40:43.386 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T12:40:43.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182477 -lt 107374182477 2024-05-15T12:40:43.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:43.797 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-365072220170 2024-05-15T12:40:43.798 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:43.799 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-15T12:40:43.799 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-365072220170 2024-05-15T12:40:43.800 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:43.801 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220170 2024-05-15T12:40:43.801 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 365072220170' 2024-05-15T12:40:43.802 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.5 seq 365072220170 2024-05-15T12:40:43.802 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-15T12:40:44.216 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220171 -lt 365072220170 2024-05-15T12:40:44.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:44.217 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-438086664195 2024-05-15T12:40:44.217 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:44.218 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-15T12:40:44.218 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-438086664195 2024-05-15T12:40:44.218 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:44.220 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664195 2024-05-15T12:40:44.220 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 438086664195' 2024-05-15T12:40:44.220 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.6 seq 438086664195 2024-05-15T12:40:44.220 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-15T12:40:44.642 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664196 -lt 438086664195 2024-05-15T12:40:44.642 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:44.643 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724601 2024-05-15T12:40:44.643 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:44.644 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-05-15T12:40:44.645 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724601 2024-05-15T12:40:44.645 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:44.646 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724601 2024-05-15T12:40:44.647 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724601' 2024-05-15T12:40:44.647 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.7 seq 167503724601 2024-05-15T12:40:44.647 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-05-15T12:40:45.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724602 -lt 167503724601 2024-05-15T12:40:45.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:45.087 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561071 2024-05-15T12:40:45.087 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:45.089 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-15T12:40:45.089 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561071 2024-05-15T12:40:45.090 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:45.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561071 2024-05-15T12:40:45.091 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561071' 2024-05-15T12:40:45.091 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.8 seq 188978561071 2024-05-15T12:40:45.091 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-15T12:40:45.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561071 -lt 188978561071 2024-05-15T12:40:45.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:45.489 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:45.490 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-210453397545 2024-05-15T12:40:45.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-05-15T12:40:45.491 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-210453397545 2024-05-15T12:40:45.491 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:45.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397545 2024-05-15T12:40:45.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 210453397545' 2024-05-15T12:40:45.493 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.9 seq 210453397545 2024-05-15T12:40:45.493 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-05-15T12:40:45.894 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397546 -lt 210453397545 2024-05-15T12:40:45.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:40:45.895 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-231928234014 2024-05-15T12:40:45.895 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:40:45.897 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-05-15T12:40:45.897 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-231928234014 2024-05-15T12:40:45.897 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:40:45.898 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928234014 2024-05-15T12:40:45.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 231928234014' 2024-05-15T12:40:45.899 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.10 seq 231928234014 2024-05-15T12:40:45.899 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-05-15T12:40:46.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928234014 -lt 231928234014 2024-05-15T12:40:46.335 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T12:40:46.335 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:40:46.335 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:40:46.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-05-15T12:40:46.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T12:40:46.912 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T12:40:46.912 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T12:40:46.913 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T12:40:46.913 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T12:40:46.913 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T12:40:46.913 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T12:40:47.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-05-15T12:40:47.327 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T12:40:47.327 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:40:47.328 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:40:47.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-05-15T12:40:47.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T12:40:47.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T12:40:47.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: grep --quiet SECONDremap-pool 2024-05-15T12:40:47.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: objectstore_tool td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-05-15T12:40:47.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-05-15T12:40:47.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-15T12:40:47.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-05-15T12:40:47.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-15T12:40:47.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-05-15T12:40:47.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-05-15T12:40:47.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-15T12:40:47.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-05-15T12:40:47.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-15T12:40:47.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.9 2024-05-15T12:40:47.872 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:40:47.872 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:40:47.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:40:47.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:40:47.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:40:49.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:40:49.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-05-15T12:40:49.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-05-15T12:40:49.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-15T12:40:49.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-05-15T12:40:49.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-15T12:40:49.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/9 2024-05-15T12:40:49.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/9 SOMETHINGremap-pool get-bytes 2024-05-15T12:40:51.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 9 2024-05-15T12:40:51.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-05-15T12:40:51.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-15T12:40:51.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-05-15T12:40:51.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-15T12:40:51.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/9 2024-05-15T12:40:51.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-15T12:40:51.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:40:51.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:40:51.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:40:51.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2024-05-15T12:40:51.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2024-05-15T12:40:51.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-15T12:40:51.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-15T12:40:51.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-15T12:40:51.881 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-15T12:40:51.881 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:40:51.881 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:40:51.881 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:40:51.881 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:40:51.881 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:40:51.883 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:40:51.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' 2024-05-15T12:40:51.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:40:51.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-15T12:40:51.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-15T12:40:51.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:40:51.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:40:51.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:40:51.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:40:51.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-15T12:40:51.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-15T12:40:51.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/9 2024-05-15T12:40:51.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-05-15T12:40:51.884 INFO:tasks.workunit.client.0.smithi005.stderr:start osd.9 2024-05-15T12:40:51.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=b169c6eb-352e-4c74-b6d0-225b04f4c043 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48542/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:40:51.885 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/9/whoami 2024-05-15T12:40:51.885 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-05-15T12:40:51.885 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-15T12:40:51.885 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-15T12:40:51.885 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-15T12:40:51.919 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:40:51.931+0000 7fa287cfc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:40:51.922 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:40:51.935+0000 7fa287cfc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:40:51.924 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:40:51.935+0000 7fa287cfc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:40:52.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-05-15T12:40:52.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:40:52.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-05-15T12:40:52.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:40:52.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:40:52.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:40:52.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:40:52.315 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T12:40:52.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:40:52.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:40:52.498 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:40:52.511+0000 7fa287cfc7c0 -1 Falling back to public interface 2024-05-15T12:40:52.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:40:53.752 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:40:53.752 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:40:53.752 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:40:53.752 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:40:53.752 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:40:53.753 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:40:54.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:40:55.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:40:55.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:40:55.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:40:55.185 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T12:40:55.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:40:55.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:40:55.399 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:40:55.411+0000 7fa287cfc7c0 -1 osd.9 105 log_to_monitors true 2024-05-15T12:40:55.687 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:40:56.688 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:40:56.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:40:56.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:40:56.689 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:40:56.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:40:56.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-15T12:40:57.450 INFO:tasks.workunit.client.0.smithi005.stderr:osd.9 up in weight 1 up_from 110 up_thru 67 down_at 106 last_clean_interval [49,105) [v2:127.0.0.1:6874/548752538,v1:127.0.0.1:6875/548752538] [v2:127.0.0.1:6876/548752538,v1:127.0.0.1:6877/548752538] exists,up fb772ddb-911d-4848-a049-c1fce0d43178 2024-05-15T12:40:57.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:40:57.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:40:57.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:40:57.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-15T12:40:57.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T12:40:57.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T12:40:57.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T12:40:57.450 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T12:40:57.451 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T12:40:57.451 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T12:40:57.451 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T12:40:57.451 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T12:40:57.451 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T12:40:57.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T12:40:57.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T12:40:57.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T12:40:57.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T12:40:57.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T12:40:57.570 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T12:40:58.617 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T12:40:58.617 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:40:58.617 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T12:40:58.617 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:40:58.617 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T12:40:58.617 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T12:40:58.617 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T12:40:58.617 INFO:tasks.workunit.client.0.smithi005.stderr:7 2024-05-15T12:40:58.618 INFO:tasks.workunit.client.0.smithi005.stderr:8 2024-05-15T12:40:58.618 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:40:58.618 INFO:tasks.workunit.client.0.smithi005.stderr:10' 2024-05-15T12:40:58.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T12:40:58.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:58.618 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T12:40:58.811 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836590 2024-05-15T12:40:58.811 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836590 2024-05-15T12:40:58.811 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590' 2024-05-15T12:40:58.811 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:58.811 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T12:40:59.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673065 2024-05-15T12:40:59.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673065 2024-05-15T12:40:59.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673065' 2024-05-15T12:40:59.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:59.273 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T12:40:59.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509536 2024-05-15T12:40:59.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509536 2024-05-15T12:40:59.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673065 2-64424509536' 2024-05-15T12:40:59.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:59.455 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T12:40:59.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449110 2024-05-15T12:40:59.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449110 2024-05-15T12:40:59.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673065 2-64424509536 3-335007449110' 2024-05-15T12:40:59.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:59.614 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T12:40:59.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182483 2024-05-15T12:40:59.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182483 2024-05-15T12:40:59.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673065 2-64424509536 3-335007449110 4-107374182483' 2024-05-15T12:40:59.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:59.766 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-15T12:40:59.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220176 2024-05-15T12:40:59.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220176 2024-05-15T12:40:59.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673065 2-64424509536 3-335007449110 4-107374182483 5-365072220176' 2024-05-15T12:40:59.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:40:59.922 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-15T12:41:00.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664201 2024-05-15T12:41:00.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664201 2024-05-15T12:41:00.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673065 2-64424509536 3-335007449110 4-107374182483 5-365072220176 6-438086664201' 2024-05-15T12:41:00.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:41:00.077 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-15T12:41:00.230 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724607 2024-05-15T12:41:00.230 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724607 2024-05-15T12:41:00.230 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673065 2-64424509536 3-335007449110 4-107374182483 5-365072220176 6-438086664201 7-167503724607' 2024-05-15T12:41:00.230 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:41:00.230 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-15T12:41:00.386 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561077 2024-05-15T12:41:00.386 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561077 2024-05-15T12:41:00.386 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673065 2-64424509536 3-335007449110 4-107374182483 5-365072220176 6-438086664201 7-167503724607 8-188978561077' 2024-05-15T12:41:00.386 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:41:00.386 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-05-15T12:41:00.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=472446402563 2024-05-15T12:41:00.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 472446402563 2024-05-15T12:41:00.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673065 2-64424509536 3-335007449110 4-107374182483 5-365072220176 6-438086664201 7-167503724607 8-188978561077 9-472446402563' 2024-05-15T12:41:00.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:41:00.541 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-05-15T12:41:00.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928234019 2024-05-15T12:41:00.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928234019 2024-05-15T12:41:00.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673065 2-64424509536 3-335007449110 4-107374182483 5-365072220176 6-438086664201 7-167503724607 8-188978561077 9-472446402563 10-231928234019' 2024-05-15T12:41:00.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:00.697 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836590 2024-05-15T12:41:00.697 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:00.698 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T12:41:00.698 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836590 2024-05-15T12:41:00.699 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:00.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836590 2024-05-15T12:41:00.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836590' 2024-05-15T12:41:00.700 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.0 seq 21474836590 2024-05-15T12:41:00.701 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T12:41:01.146 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836590 -lt 21474836590 2024-05-15T12:41:01.146 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:01.147 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673065 2024-05-15T12:41:01.147 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:01.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T12:41:01.149 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673065 2024-05-15T12:41:01.150 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:01.151 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673065 2024-05-15T12:41:01.151 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673065' 2024-05-15T12:41:01.151 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.1 seq 42949673065 2024-05-15T12:41:01.151 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T12:41:01.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673066 -lt 42949673065 2024-05-15T12:41:01.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:01.580 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509536 2024-05-15T12:41:01.580 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:01.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T12:41:01.582 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509536 2024-05-15T12:41:01.582 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:01.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509536 2024-05-15T12:41:01.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-05-15T12:41:01.584 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.2 seq 64424509536 2024-05-15T12:41:01.584 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T12:41:02.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509536 -lt 64424509536 2024-05-15T12:41:02.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:02.058 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-335007449110 2024-05-15T12:41:02.058 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:02.060 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T12:41:02.061 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:02.061 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-335007449110 2024-05-15T12:41:02.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449110 2024-05-15T12:41:02.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 335007449110' 2024-05-15T12:41:02.062 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.3 seq 335007449110 2024-05-15T12:41:02.062 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T12:41:02.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449110 -lt 335007449110 2024-05-15T12:41:02.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:02.496 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182483 2024-05-15T12:41:02.496 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:02.498 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T12:41:02.498 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182483 2024-05-15T12:41:02.498 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:02.500 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182483 2024-05-15T12:41:02.500 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182483' 2024-05-15T12:41:02.500 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.4 seq 107374182483 2024-05-15T12:41:02.500 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T12:41:02.948 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182483 -lt 107374182483 2024-05-15T12:41:02.948 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:02.949 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-365072220176 2024-05-15T12:41:02.949 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:02.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-15T12:41:02.951 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-365072220176 2024-05-15T12:41:02.952 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:02.954 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220176 2024-05-15T12:41:02.954 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 365072220176' 2024-05-15T12:41:02.954 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.5 seq 365072220176 2024-05-15T12:41:02.954 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-15T12:41:03.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220177 -lt 365072220176 2024-05-15T12:41:03.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:03.402 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-438086664201 2024-05-15T12:41:03.402 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:03.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-15T12:41:03.404 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-438086664201 2024-05-15T12:41:03.404 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:03.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664201 2024-05-15T12:41:03.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 438086664201' 2024-05-15T12:41:03.405 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.6 seq 438086664201 2024-05-15T12:41:03.406 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-15T12:41:03.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664202 -lt 438086664201 2024-05-15T12:41:03.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:03.830 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724607 2024-05-15T12:41:03.830 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:03.832 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-05-15T12:41:03.833 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724607 2024-05-15T12:41:03.833 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:03.835 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724607 2024-05-15T12:41:03.835 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724607' 2024-05-15T12:41:03.835 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.7 seq 167503724607 2024-05-15T12:41:03.835 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-05-15T12:41:04.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724607 -lt 167503724607 2024-05-15T12:41:04.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:04.281 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:04.281 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561077 2024-05-15T12:41:04.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-15T12:41:04.283 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561077 2024-05-15T12:41:04.283 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:04.284 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561077 2024-05-15T12:41:04.284 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561077' 2024-05-15T12:41:04.284 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.8 seq 188978561077 2024-05-15T12:41:04.285 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-15T12:41:04.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561077 -lt 188978561077 2024-05-15T12:41:04.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:04.706 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-472446402563 2024-05-15T12:41:04.706 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:04.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-05-15T12:41:04.708 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-472446402563 2024-05-15T12:41:04.708 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:04.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=472446402563 2024-05-15T12:41:04.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 472446402563' 2024-05-15T12:41:04.709 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.9 seq 472446402563 2024-05-15T12:41:04.709 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-05-15T12:41:05.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 472446402564 -lt 472446402563 2024-05-15T12:41:05.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:05.156 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-231928234019 2024-05-15T12:41:05.156 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:05.157 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-05-15T12:41:05.158 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-231928234019 2024-05-15T12:41:05.158 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:05.159 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928234019 2024-05-15T12:41:05.159 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 231928234019' 2024-05-15T12:41:05.159 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.10 seq 231928234019 2024-05-15T12:41:05.160 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-05-15T12:41:05.574 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928234020 -lt 231928234019 2024-05-15T12:41:05.574 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T12:41:05.575 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:41:05.575 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:41:06.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-05-15T12:41:06.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T12:41:06.155 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T12:41:06.155 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T12:41:06.155 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T12:41:06.155 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T12:41:06.156 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T12:41:06.156 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T12:41:06.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-05-15T12:41:06.612 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T12:41:06.612 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:41:06.613 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:41:07.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-05-15T12:41:07.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T12:41:07.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T12:41:07.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:329: TEST_chunk_mapping: delete_pool remap-pool 2024-05-15T12:41:07.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=remap-pool 2024-05-15T12:41:07.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete remap-pool remap-pool --yes-i-really-really-mean-it 2024-05-15T12:41:07.687 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'remap-pool' does not exist 2024-05-15T12:41:07.702 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:330: TEST_chunk_mapping: ceph osd erasure-code-profile rm remap-profile 2024-05-15T12:41:08.464 INFO:tasks.workunit.client.0.smithi005.stderr:erasure-code-profile remap-profile does not exist 2024-05-15T12:41:08.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-05-15T12:41:08.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_isa td/test-erasure-code 2024-05-15T12:41:08.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:193: TEST_rados_put_get_isa: erasure_code_plugin_exists isa 2024-05-15T12:41:08.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: erasure_code_plugin_exists: local plugin=isa 2024-05-15T12:41:08.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: erasure_code_plugin_exists: local status 2024-05-15T12:41:08.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2053: erasure_code_plugin_exists: local grepstr 2024-05-15T12:41:08.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2054: erasure_code_plugin_exists: local s 2024-05-15T12:41:08.482 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2055: erasure_code_plugin_exists: case `uname` in 2024-05-15T12:41:08.482 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2055: erasure_code_plugin_exists: uname 2024-05-15T12:41:08.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: erasure_code_plugin_exists: grepstr='isa.*No such file' 2024-05-15T12:41:08.483 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=isa 2024-05-15T12:41:09.248 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: s= 2024-05-15T12:41:09.248 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: erasure_code_plugin_exists: local status=0 2024-05-15T12:41:09.249 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2062: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-05-15T12:41:09.249 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-05-15T12:41:10.311 INFO:tasks.workunit.client.0.smithi005.stderr:erasure-code-profile TESTPROFILE does not exist 2024-05-15T12:41:10.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: return 0 2024-05-15T12:41:10.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:197: TEST_rados_put_get_isa: local dir=td/test-erasure-code 2024-05-15T12:41:10.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:198: TEST_rados_put_get_isa: local poolname=pool-isa 2024-05-15T12:41:10.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:200: TEST_rados_put_get_isa: ceph osd erasure-code-profile set profile-isa plugin=isa crush-failure-domain=osd 2024-05-15T12:41:11.167 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:203: TEST_rados_put_get_isa: create_pool pool-isa 1 1 erasure profile-isa 2024-05-15T12:41:11.168 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-isa 1 1 erasure profile-isa 2024-05-15T12:41:12.182 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-isa' already exists 2024-05-15T12:41:12.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T12:41:13.203 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:206: TEST_rados_put_get_isa: rados_put_get td/test-erasure-code pool-isa 2024-05-15T12:41:13.203 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-05-15T12:41:13.203 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-isa 2024-05-15T12:41:13.203 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-05-15T12:41:13.203 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:41:13.203 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-05-15T12:41:13.203 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:41:13.203 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-05-15T12:41:13.204 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:41:13.204 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-05-15T12:41:13.204 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:41:13.204 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-05-15T12:41:13.204 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-isa put SOMETHING td/test-erasure-code/ORIGINAL 2024-05-15T12:41:15.070 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2024-05-15T12:41:15.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-15T12:41:15.417 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-05-15T12:41:15.420 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-isa SOMETHING 2024-05-15T12:41:15.420 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-isa 2024-05-15T12:41:15.420 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-15T12:41:15.421 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-isa SOMETHING 2024-05-15T12:41:15.421 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:41:15.871 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-05-15T12:41:15.871 INFO:tasks.workunit.client.0.smithi005.stderr:10 2024-05-15T12:41:15.871 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T12:41:15.871 INFO:tasks.workunit.client.0.smithi005.stderr:7 2024-05-15T12:41:15.871 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T12:41:15.871 INFO:tasks.workunit.client.0.smithi005.stderr:8 2024-05-15T12:41:15.871 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T12:41:15.871 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:41:15.871 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T12:41:15.872 INFO:tasks.workunit.client.0.smithi005.stderr:9' 2024-05-15T12:41:15.872 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 10 0 7 6 8 2 1 5 9 2024-05-15T12:41:15.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('4' '10' '0' '7' '6' '8' '2' '1' '5' '9') 2024-05-15T12:41:15.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-05-15T12:41:15.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=9 2024-05-15T12:41:15.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2024-05-15T12:41:16.425 INFO:tasks.workunit.client.0.smithi005.stderr:osd.9 is already out. 2024-05-15T12:41:16.442 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-05-15T12:41:21.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-isa SOMETHING 2024-05-15T12:41:21.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<9\>' 2024-05-15T12:41:21.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-isa 2024-05-15T12:41:21.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-15T12:41:21.447 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-isa SOMETHING 2024-05-15T12:41:21.447 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:41:21.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-05-15T12:41:21.882 INFO:tasks.workunit.client.0.smithi005.stderr:10 2024-05-15T12:41:21.882 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T12:41:21.883 INFO:tasks.workunit.client.0.smithi005.stderr:7 2024-05-15T12:41:21.883 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T12:41:21.883 INFO:tasks.workunit.client.0.smithi005.stderr:8 2024-05-15T12:41:21.883 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:41:21.883 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:41:21.883 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T12:41:21.883 INFO:tasks.workunit.client.0.smithi005.stderr:2' 2024-05-15T12:41:21.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 10 0 7 6 8 3 1 5 2 2024-05-15T12:41:21.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2024-05-15T12:41:21.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-15T12:41:21.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 9 2024-05-15T12:41:22.475 INFO:tasks.workunit.client.0.smithi005.stderr:osd.9 is already in. 2024-05-15T12:41:22.493 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-05-15T12:41:22.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:208: TEST_rados_put_get_isa: delete_pool pool-isa 2024-05-15T12:41:22.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-isa 2024-05-15T12:41:22.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-isa pool-isa --yes-i-really-really-mean-it 2024-05-15T12:41:24.437 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-isa' does not exist 2024-05-15T12:41:24.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-05-15T12:41:24.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_jerasure td/test-erasure-code 2024-05-15T12:41:24.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:212: TEST_rados_put_get_jerasure: local dir=td/test-erasure-code 2024-05-15T12:41:24.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:214: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code ecpool 2024-05-15T12:41:24.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-05-15T12:41:24.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=ecpool 2024-05-15T12:41:24.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-05-15T12:41:24.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:41:24.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-05-15T12:41:24.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:41:24.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-05-15T12:41:24.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:41:24.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-05-15T12:41:24.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:41:24.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-05-15T12:41:24.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool ecpool put SOMETHING td/test-erasure-code/ORIGINAL 2024-05-15T12:41:25.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2024-05-15T12:41:25.901 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-15T12:41:25.903 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-05-15T12:41:25.938 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds ecpool SOMETHING 2024-05-15T12:41:25.938 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-05-15T12:41:25.938 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-15T12:41:25.939 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-05-15T12:41:25.939 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:41:26.460 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=6 2024-05-15T12:41:26.460 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T12:41:26.460 INFO:tasks.workunit.client.0.smithi005.stderr:7 2024-05-15T12:41:26.460 INFO:tasks.workunit.client.0.smithi005.stderr:9' 2024-05-15T12:41:26.460 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 6 0 7 9 2024-05-15T12:41:26.460 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('6' '0' '7' '9') 2024-05-15T12:41:26.460 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-05-15T12:41:26.460 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=3 2024-05-15T12:41:26.460 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2024-05-15T12:41:27.089 INFO:tasks.workunit.client.0.smithi005.stderr:osd.9 is already out. 2024-05-15T12:41:27.111 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-05-15T12:41:32.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds ecpool SOMETHING 2024-05-15T12:41:32.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-05-15T12:41:32.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-15T12:41:32.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<9\>' 2024-05-15T12:41:32.114 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:41:32.115 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-05-15T12:41:32.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=6 2024-05-15T12:41:32.586 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T12:41:32.586 INFO:tasks.workunit.client.0.smithi005.stderr:7 2024-05-15T12:41:32.586 INFO:tasks.workunit.client.0.smithi005.stderr:5' 2024-05-15T12:41:32.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 6 0 7 5 2024-05-15T12:41:32.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2024-05-15T12:41:32.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-15T12:41:32.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 9 2024-05-15T12:41:33.219 INFO:tasks.workunit.client.0.smithi005.stderr:osd.9 is already in. 2024-05-15T12:41:33.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-05-15T12:41:33.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:216: TEST_rados_put_get_jerasure: local poolname=pool-jerasure 2024-05-15T12:41:33.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:217: TEST_rados_put_get_jerasure: local profile=profile-jerasure 2024-05-15T12:41:33.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:219: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile set profile-jerasure plugin=jerasure k=4 m=2 crush-failure-domain=osd 2024-05-15T12:41:35.120 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:223: TEST_rados_put_get_jerasure: create_pool pool-jerasure 12 12 erasure profile-jerasure 2024-05-15T12:41:35.120 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 12 12 erasure profile-jerasure 2024-05-15T12:41:36.807 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-jerasure' already exists 2024-05-15T12:41:36.825 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T12:41:37.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:226: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code pool-jerasure 2024-05-15T12:41:37.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-05-15T12:41:37.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-jerasure 2024-05-15T12:41:37.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-05-15T12:41:37.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:41:37.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-05-15T12:41:37.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:41:37.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-05-15T12:41:37.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:41:37.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-05-15T12:41:37.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:41:37.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-05-15T12:41:37.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2024-05-15T12:41:39.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-05-15T12:41:39.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-15T12:41:39.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-05-15T12:41:39.664 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-jerasure SOMETHING 2024-05-15T12:41:39.664 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T12:41:39.664 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-15T12:41:39.665 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-05-15T12:41:39.665 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:41:40.571 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-05-15T12:41:40.571 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:41:40.571 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:41:40.571 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T12:41:40.571 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:41:40.571 INFO:tasks.workunit.client.0.smithi005.stderr:10' 2024-05-15T12:41:40.571 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-05-15T12:41:40.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '1' '9' '4' '3' '10') 2024-05-15T12:41:40.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-05-15T12:41:40.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=5 2024-05-15T12:41:40.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 10 2024-05-15T12:41:41.268 INFO:tasks.workunit.client.0.smithi005.stderr:osd.10 is already out. 2024-05-15T12:41:41.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-05-15T12:41:46.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-jerasure SOMETHING 2024-05-15T12:41:46.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T12:41:46.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<10\>' 2024-05-15T12:41:46.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-15T12:41:46.289 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-05-15T12:41:46.289 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:41:46.716 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-05-15T12:41:46.717 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:41:46.717 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:41:46.717 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T12:41:46.717 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:41:46.717 INFO:tasks.workunit.client.0.smithi005.stderr:6' 2024-05-15T12:41:46.717 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 6 2024-05-15T12:41:46.717 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-05-15T12:41:46.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-15T12:41:46.806 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 10 2024-05-15T12:41:47.380 INFO:tasks.workunit.client.0.smithi005.stderr:osd.10 is already in. 2024-05-15T12:41:47.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-05-15T12:41:47.398 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:227: TEST_rados_put_get_jerasure: rados_osds_out_in td/test-erasure-code pool-jerasure 2024-05-15T12:41:47.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:104: rados_osds_out_in: local dir=td/test-erasure-code 2024-05-15T12:41:47.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: local poolname=pool-jerasure 2024-05-15T12:41:47.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: local objname=SOMETHING 2024-05-15T12:41:47.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-05-15T12:41:47.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 FFFF 2024-05-15T12:41:47.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-05-15T12:41:47.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 GGGG 2024-05-15T12:41:47.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-05-15T12:41:47.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 HHHH 2024-05-15T12:41:47.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-05-15T12:41:47.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 IIII 2024-05-15T12:41:47.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:116: rados_osds_out_in: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2024-05-15T12:41:50.903 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:117: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-05-15T12:41:50.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:118: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-15T12:41:50.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:119: rados_osds_out_in: rm td/test-erasure-code/COPY 2024-05-15T12:41:50.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:127: rados_osds_out_in: wait_for_clean 2024-05-15T12:41:50.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T12:41:50.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T12:41:50.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T12:41:51.000 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T12:41:51.001 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T12:41:51.001 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T12:41:51.001 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T12:41:51.001 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T12:41:51.001 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T12:41:51.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T12:41:51.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T12:41:51.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T12:41:51.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T12:41:51.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T12:41:51.129 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T12:41:51.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T12:41:51.585 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:41:51.586 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T12:41:51.586 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:41:51.586 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T12:41:51.586 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T12:41:51.586 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T12:41:51.586 INFO:tasks.workunit.client.0.smithi005.stderr:7 2024-05-15T12:41:51.586 INFO:tasks.workunit.client.0.smithi005.stderr:8 2024-05-15T12:41:51.586 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:41:51.586 INFO:tasks.workunit.client.0.smithi005.stderr:10' 2024-05-15T12:41:51.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T12:41:51.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:41:51.587 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T12:41:51.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836602 2024-05-15T12:41:51.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836602 2024-05-15T12:41:51.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602' 2024-05-15T12:41:51.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:41:51.748 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T12:41:51.902 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673078 2024-05-15T12:41:51.902 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673078 2024-05-15T12:41:51.902 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673078' 2024-05-15T12:41:51.902 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:41:51.903 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T12:41:52.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509549 2024-05-15T12:41:52.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509549 2024-05-15T12:41:52.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673078 2-64424509549' 2024-05-15T12:41:52.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:41:52.064 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T12:41:52.218 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449122 2024-05-15T12:41:52.218 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449122 2024-05-15T12:41:52.219 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673078 2-64424509549 3-335007449122' 2024-05-15T12:41:52.219 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:41:52.219 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T12:41:52.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182495 2024-05-15T12:41:52.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182495 2024-05-15T12:41:52.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673078 2-64424509549 3-335007449122 4-107374182495' 2024-05-15T12:41:52.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:41:52.378 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-15T12:41:52.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220189 2024-05-15T12:41:52.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220189 2024-05-15T12:41:52.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673078 2-64424509549 3-335007449122 4-107374182495 5-365072220189' 2024-05-15T12:41:52.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:41:52.536 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-15T12:41:52.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664214 2024-05-15T12:41:52.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664214 2024-05-15T12:41:52.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673078 2-64424509549 3-335007449122 4-107374182495 5-365072220189 6-438086664214' 2024-05-15T12:41:52.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:41:52.690 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-15T12:41:52.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724619 2024-05-15T12:41:52.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724619 2024-05-15T12:41:52.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673078 2-64424509549 3-335007449122 4-107374182495 5-365072220189 6-438086664214 7-167503724619' 2024-05-15T12:41:52.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:41:52.834 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-15T12:41:52.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561089 2024-05-15T12:41:52.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561089 2024-05-15T12:41:52.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673078 2-64424509549 3-335007449122 4-107374182495 5-365072220189 6-438086664214 7-167503724619 8-188978561089' 2024-05-15T12:41:52.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:41:52.980 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-05-15T12:41:53.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=472446402576 2024-05-15T12:41:53.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 472446402576 2024-05-15T12:41:53.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673078 2-64424509549 3-335007449122 4-107374182495 5-365072220189 6-438086664214 7-167503724619 8-188978561089 9-472446402576' 2024-05-15T12:41:53.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:41:53.126 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-05-15T12:41:53.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928234032 2024-05-15T12:41:53.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928234032 2024-05-15T12:41:53.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673078 2-64424509549 3-335007449122 4-107374182495 5-365072220189 6-438086664214 7-167503724619 8-188978561089 9-472446402576 10-231928234032' 2024-05-15T12:41:53.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:53.268 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836602 2024-05-15T12:41:53.268 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:53.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T12:41:53.269 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836602 2024-05-15T12:41:53.269 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:53.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836602 2024-05-15T12:41:53.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836602' 2024-05-15T12:41:53.271 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 21474836602 2024-05-15T12:41:53.271 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T12:41:53.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836602 -lt 21474836602 2024-05-15T12:41:53.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:53.736 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673078 2024-05-15T12:41:53.736 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:53.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T12:41:53.738 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673078 2024-05-15T12:41:53.739 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:53.740 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673078 2024-05-15T12:41:53.740 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 42949673078 2024-05-15T12:41:53.741 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673078' 2024-05-15T12:41:53.741 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T12:41:54.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673078 -lt 42949673078 2024-05-15T12:41:54.161 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:54.161 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509549 2024-05-15T12:41:54.161 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:54.162 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T12:41:54.163 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509549 2024-05-15T12:41:54.163 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:54.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509549 2024-05-15T12:41:54.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509549' 2024-05-15T12:41:54.165 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 64424509549 2024-05-15T12:41:54.165 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T12:41:54.581 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509549 -lt 64424509549 2024-05-15T12:41:54.581 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:54.582 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-335007449122 2024-05-15T12:41:54.583 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:54.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T12:41:54.585 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-335007449122 2024-05-15T12:41:54.585 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:54.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449122 2024-05-15T12:41:54.587 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 335007449122 2024-05-15T12:41:54.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 335007449122' 2024-05-15T12:41:54.588 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T12:41:55.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449123 -lt 335007449122 2024-05-15T12:41:55.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:55.038 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182495 2024-05-15T12:41:55.038 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:55.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T12:41:55.040 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182495 2024-05-15T12:41:55.040 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:55.041 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182495 2024-05-15T12:41:55.042 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.4 seq 107374182495 2024-05-15T12:41:55.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182495' 2024-05-15T12:41:55.042 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T12:41:55.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182496 -lt 107374182495 2024-05-15T12:41:55.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:55.588 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-365072220189 2024-05-15T12:41:55.588 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:55.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-15T12:41:55.591 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-365072220189 2024-05-15T12:41:55.591 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:55.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220189 2024-05-15T12:41:55.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 365072220189' 2024-05-15T12:41:55.593 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.5 seq 365072220189 2024-05-15T12:41:55.593 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-15T12:41:56.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220189 -lt 365072220189 2024-05-15T12:41:56.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:56.035 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-438086664214 2024-05-15T12:41:56.035 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:56.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-15T12:41:56.038 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-438086664214 2024-05-15T12:41:56.038 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:56.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664214 2024-05-15T12:41:56.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 438086664214' 2024-05-15T12:41:56.039 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.6 seq 438086664214 2024-05-15T12:41:56.040 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-15T12:41:56.500 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664214 -lt 438086664214 2024-05-15T12:41:56.500 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:56.501 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724619 2024-05-15T12:41:56.501 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:56.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-05-15T12:41:56.503 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724619 2024-05-15T12:41:56.503 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:56.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724619 2024-05-15T12:41:56.505 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.7 seq 167503724619 2024-05-15T12:41:56.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724619' 2024-05-15T12:41:56.505 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-05-15T12:41:56.991 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724620 -lt 167503724619 2024-05-15T12:41:56.991 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:56.992 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561089 2024-05-15T12:41:56.992 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:56.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-15T12:41:56.995 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561089 2024-05-15T12:41:56.995 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:56.996 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561089 2024-05-15T12:41:56.996 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.8 seq 188978561089 2024-05-15T12:41:56.996 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561089' 2024-05-15T12:41:56.997 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-15T12:41:57.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561090 -lt 188978561089 2024-05-15T12:41:57.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:57.427 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-472446402576 2024-05-15T12:41:57.428 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:57.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-05-15T12:41:57.429 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-472446402576 2024-05-15T12:41:57.430 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:57.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=472446402576 2024-05-15T12:41:57.431 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.9 seq 472446402576 2024-05-15T12:41:57.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 472446402576' 2024-05-15T12:41:57.432 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-05-15T12:41:57.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 472446402577 -lt 472446402576 2024-05-15T12:41:57.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:41:57.963 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-231928234032 2024-05-15T12:41:57.963 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:41:57.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-05-15T12:41:57.965 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-231928234032 2024-05-15T12:41:57.965 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:41:57.965 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928234032 2024-05-15T12:41:57.966 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.10 seq 231928234032 2024-05-15T12:41:57.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 231928234032' 2024-05-15T12:41:57.967 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-05-15T12:41:58.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928234033 -lt 231928234032 2024-05-15T12:41:58.450 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T12:41:58.450 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:41:58.450 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:41:58.984 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-05-15T12:41:58.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T12:41:58.986 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T12:41:58.986 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T12:41:58.986 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T12:41:58.986 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T12:41:58.986 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T12:41:58.986 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T12:41:59.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-05-15T12:41:59.422 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T12:41:59.423 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:41:59.423 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:41:59.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-05-15T12:41:59.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T12:41:59.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T12:41:59.967 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-05-15T12:41:59.967 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T12:41:59.967 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-15T12:41:59.968 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:41:59.968 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-05-15T12:42:00.458 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-05-15T12:42:00.458 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:42:00.458 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:42:00.458 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T12:42:00.458 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:42:00.458 INFO:tasks.workunit.client.0.smithi005.stderr:10' 2024-05-15T12:42:00.458 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-05-15T12:42:00.458 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: local 'osds_list=5 1 9 4 3 10' 2024-05-15T12:42:00.459 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: osds=('5' '1' '9' '4' '3' '10') 2024-05-15T12:42:00.459 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: local -a osds 2024-05-15T12:42:00.459 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2024-05-15T12:42:00.459 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 5 2024-05-15T12:42:01.034 INFO:tasks.workunit.client.0.smithi005.stderr:osd.5 is already out. 2024-05-15T12:42:01.051 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2024-05-15T12:42:01.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 1 2024-05-15T12:42:02.655 INFO:tasks.workunit.client.0.smithi005.stderr:osd.1 is already out. 2024-05-15T12:42:02.677 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:133: rados_osds_out_in: wait_for_clean 2024-05-15T12:42:02.677 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T12:42:02.678 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T12:42:02.678 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T12:42:02.678 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T12:42:02.679 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T12:42:02.679 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T12:42:02.679 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T12:42:02.680 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T12:42:02.680 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T12:42:02.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T12:42:02.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T12:42:02.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T12:42:02.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T12:42:02.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T12:42:02.795 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T12:42:03.808 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T12:42:03.808 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:42:03.808 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T12:42:03.808 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:42:03.808 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T12:42:03.808 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T12:42:03.808 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T12:42:03.808 INFO:tasks.workunit.client.0.smithi005.stderr:7 2024-05-15T12:42:03.808 INFO:tasks.workunit.client.0.smithi005.stderr:8 2024-05-15T12:42:03.808 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:42:03.808 INFO:tasks.workunit.client.0.smithi005.stderr:10' 2024-05-15T12:42:03.808 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T12:42:03.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:03.809 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T12:42:04.240 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836607 2024-05-15T12:42:04.240 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836607 2024-05-15T12:42:04.240 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607' 2024-05-15T12:42:04.240 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:04.240 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T12:42:04.576 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673082 2024-05-15T12:42:04.576 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673082 2024-05-15T12:42:04.576 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673082' 2024-05-15T12:42:04.576 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:04.577 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T12:42:05.187 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509553 2024-05-15T12:42:05.187 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509553 2024-05-15T12:42:05.187 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673082 2-64424509553' 2024-05-15T12:42:05.187 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:05.187 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T12:42:05.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449127 2024-05-15T12:42:05.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449127 2024-05-15T12:42:05.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673082 2-64424509553 3-335007449127' 2024-05-15T12:42:05.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:05.486 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T12:42:05.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182500 2024-05-15T12:42:05.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182500 2024-05-15T12:42:05.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673082 2-64424509553 3-335007449127 4-107374182500' 2024-05-15T12:42:05.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:05.975 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-15T12:42:06.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220194 2024-05-15T12:42:06.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220194 2024-05-15T12:42:06.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673082 2-64424509553 3-335007449127 4-107374182500 5-365072220194' 2024-05-15T12:42:06.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:06.135 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-15T12:42:06.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664218 2024-05-15T12:42:06.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664218 2024-05-15T12:42:06.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673082 2-64424509553 3-335007449127 4-107374182500 5-365072220194 6-438086664218' 2024-05-15T12:42:06.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:06.291 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-15T12:42:06.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724624 2024-05-15T12:42:06.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724624 2024-05-15T12:42:06.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673082 2-64424509553 3-335007449127 4-107374182500 5-365072220194 6-438086664218 7-167503724624' 2024-05-15T12:42:06.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:06.447 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-15T12:42:06.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561094 2024-05-15T12:42:06.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561094 2024-05-15T12:42:06.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673082 2-64424509553 3-335007449127 4-107374182500 5-365072220194 6-438086664218 7-167503724624 8-188978561094' 2024-05-15T12:42:06.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:06.604 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-05-15T12:42:06.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=472446402581 2024-05-15T12:42:06.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 472446402581 2024-05-15T12:42:06.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673082 2-64424509553 3-335007449127 4-107374182500 5-365072220194 6-438086664218 7-167503724624 8-188978561094 9-472446402581' 2024-05-15T12:42:06.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:06.764 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-05-15T12:42:06.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928234037 2024-05-15T12:42:06.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928234037 2024-05-15T12:42:06.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673082 2-64424509553 3-335007449127 4-107374182500 5-365072220194 6-438086664218 7-167503724624 8-188978561094 9-472446402581 10-231928234037' 2024-05-15T12:42:06.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:06.919 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836607 2024-05-15T12:42:06.919 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:06.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T12:42:06.922 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836607 2024-05-15T12:42:06.922 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:06.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836607 2024-05-15T12:42:06.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836607' 2024-05-15T12:42:06.923 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 21474836607 2024-05-15T12:42:06.923 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T12:42:07.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836607 -lt 21474836607 2024-05-15T12:42:07.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:07.488 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673082 2024-05-15T12:42:07.488 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:07.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T12:42:07.490 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673082 2024-05-15T12:42:07.490 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:07.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673082 2024-05-15T12:42:07.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673082' 2024-05-15T12:42:07.492 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 42949673082 2024-05-15T12:42:07.492 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T12:42:08.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673083 -lt 42949673082 2024-05-15T12:42:08.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:08.001 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509553 2024-05-15T12:42:08.001 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:08.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T12:42:08.003 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509553 2024-05-15T12:42:08.004 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:08.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509553 2024-05-15T12:42:08.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509553' 2024-05-15T12:42:08.005 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 64424509553 2024-05-15T12:42:08.005 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T12:42:08.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509553 -lt 64424509553 2024-05-15T12:42:08.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:08.508 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-335007449127 2024-05-15T12:42:08.508 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:08.510 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T12:42:08.511 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-335007449127 2024-05-15T12:42:08.511 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:08.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449127 2024-05-15T12:42:08.513 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 335007449127 2024-05-15T12:42:08.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 335007449127' 2024-05-15T12:42:08.514 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T12:42:08.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449127 -lt 335007449127 2024-05-15T12:42:08.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:08.995 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182500 2024-05-15T12:42:08.995 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:08.996 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T12:42:08.997 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182500 2024-05-15T12:42:08.997 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:08.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182500 2024-05-15T12:42:08.999 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.4 seq 107374182500 2024-05-15T12:42:08.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182500' 2024-05-15T12:42:08.999 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T12:42:09.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182500 -lt 107374182500 2024-05-15T12:42:09.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:09.436 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-365072220194 2024-05-15T12:42:09.436 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:09.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-15T12:42:09.439 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-365072220194 2024-05-15T12:42:09.439 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:09.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220194 2024-05-15T12:42:09.440 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.5 seq 365072220194 2024-05-15T12:42:09.441 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 365072220194' 2024-05-15T12:42:09.441 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-15T12:42:09.861 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220194 -lt 365072220194 2024-05-15T12:42:09.861 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:09.861 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-438086664218 2024-05-15T12:42:09.862 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:09.864 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-15T12:42:09.864 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-438086664218 2024-05-15T12:42:09.865 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:09.865 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664218 2024-05-15T12:42:09.865 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 438086664218' 2024-05-15T12:42:09.866 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.6 seq 438086664218 2024-05-15T12:42:09.866 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-15T12:42:10.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664219 -lt 438086664218 2024-05-15T12:42:10.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:10.314 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724624 2024-05-15T12:42:10.314 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:10.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-05-15T12:42:10.316 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724624 2024-05-15T12:42:10.316 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:10.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724624 2024-05-15T12:42:10.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724624' 2024-05-15T12:42:10.318 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.7 seq 167503724624 2024-05-15T12:42:10.318 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-05-15T12:42:10.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724625 -lt 167503724624 2024-05-15T12:42:10.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:10.860 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561094 2024-05-15T12:42:10.860 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:10.862 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-15T12:42:10.862 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561094 2024-05-15T12:42:10.863 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:10.864 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561094 2024-05-15T12:42:10.865 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.8 seq 188978561094 2024-05-15T12:42:10.865 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561094' 2024-05-15T12:42:10.865 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-15T12:42:11.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561095 -lt 188978561094 2024-05-15T12:42:11.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:11.288 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-472446402581 2024-05-15T12:42:11.289 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:11.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-05-15T12:42:11.291 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:11.291 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-472446402581 2024-05-15T12:42:11.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=472446402581 2024-05-15T12:42:11.292 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.9 seq 472446402581 2024-05-15T12:42:11.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 472446402581' 2024-05-15T12:42:11.293 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-05-15T12:42:11.713 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 472446402581 -lt 472446402581 2024-05-15T12:42:11.713 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:11.714 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-231928234037 2024-05-15T12:42:11.714 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:11.716 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-05-15T12:42:11.717 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-231928234037 2024-05-15T12:42:11.718 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:11.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928234037 2024-05-15T12:42:11.719 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.10 seq 231928234037 2024-05-15T12:42:11.719 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 231928234037' 2024-05-15T12:42:11.719 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-05-15T12:42:12.166 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928234037 -lt 231928234037 2024-05-15T12:42:12.167 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T12:42:12.168 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:42:12.168 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:42:12.780 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-05-15T12:42:12.780 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T12:42:12.781 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T12:42:12.781 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T12:42:12.781 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T12:42:12.781 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T12:42:12.781 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T12:42:12.782 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T12:42:13.208 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-05-15T12:42:13.209 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T12:42:13.209 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:42:13.209 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:42:13.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-05-15T12:42:13.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T12:42:13.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T12:42:13.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2024-05-15T12:42:13.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-05-15T12:42:13.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T12:42:13.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-15T12:42:13.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<5\>' 2024-05-15T12:42:13.762 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-05-15T12:42:13.762 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:42:14.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-05-15T12:42:14.194 INFO:tasks.workunit.client.0.smithi005.stderr:7 2024-05-15T12:42:14.194 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:42:14.194 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T12:42:14.194 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:42:14.194 INFO:tasks.workunit.client.0.smithi005.stderr:10' 2024-05-15T12:42:14.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 7 9 4 3 10 2024-05-15T12:42:14.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2024-05-15T12:42:14.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-05-15T12:42:14.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T12:42:14.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-15T12:42:14.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<1\>' 2024-05-15T12:42:14.196 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-05-15T12:42:14.196 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:42:14.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-05-15T12:42:14.631 INFO:tasks.workunit.client.0.smithi005.stderr:7 2024-05-15T12:42:14.631 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:42:14.631 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T12:42:14.631 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:42:14.631 INFO:tasks.workunit.client.0.smithi005.stderr:10' 2024-05-15T12:42:14.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 7 9 4 3 10 2024-05-15T12:42:14.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:140: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-05-15T12:42:14.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:141: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-15T12:42:14.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2024-05-15T12:42:14.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 5 2024-05-15T12:42:15.285 INFO:tasks.workunit.client.0.smithi005.stderr:osd.5 is already in. 2024-05-15T12:42:15.303 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2024-05-15T12:42:15.303 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 1 2024-05-15T12:42:17.628 INFO:tasks.workunit.client.0.smithi005.stderr:osd.1 is already in. 2024-05-15T12:42:17.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:150: rados_osds_out_in: wait_for_clean 2024-05-15T12:42:17.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T12:42:17.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T12:42:17.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T12:42:17.653 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T12:42:17.653 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T12:42:17.653 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T12:42:17.654 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T12:42:17.654 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T12:42:17.654 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T12:42:17.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T12:42:17.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T12:42:17.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T12:42:17.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T12:42:17.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T12:42:17.762 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T12:42:18.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T12:42:18.517 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:42:18.517 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T12:42:18.517 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:42:18.517 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T12:42:18.518 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T12:42:18.518 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T12:42:18.518 INFO:tasks.workunit.client.0.smithi005.stderr:7 2024-05-15T12:42:18.518 INFO:tasks.workunit.client.0.smithi005.stderr:8 2024-05-15T12:42:18.518 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:42:18.518 INFO:tasks.workunit.client.0.smithi005.stderr:10' 2024-05-15T12:42:18.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T12:42:18.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:18.518 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T12:42:18.984 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836612 2024-05-15T12:42:18.984 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836612 2024-05-15T12:42:18.984 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612' 2024-05-15T12:42:18.984 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:18.985 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T12:42:19.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673087 2024-05-15T12:42:19.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673087 2024-05-15T12:42:19.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612 1-42949673087' 2024-05-15T12:42:19.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:19.229 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T12:42:19.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509558 2024-05-15T12:42:19.626 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509558 2024-05-15T12:42:19.626 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612 1-42949673087 2-64424509558' 2024-05-15T12:42:19.626 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:19.626 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T12:42:19.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449132 2024-05-15T12:42:19.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449132 2024-05-15T12:42:19.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612 1-42949673087 2-64424509558 3-335007449132' 2024-05-15T12:42:19.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:19.860 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T12:42:20.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182505 2024-05-15T12:42:20.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182505 2024-05-15T12:42:20.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612 1-42949673087 2-64424509558 3-335007449132 4-107374182505' 2024-05-15T12:42:20.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:20.298 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-15T12:42:20.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220198 2024-05-15T12:42:20.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220198 2024-05-15T12:42:20.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612 1-42949673087 2-64424509558 3-335007449132 4-107374182505 5-365072220198' 2024-05-15T12:42:20.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:20.481 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-15T12:42:20.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664223 2024-05-15T12:42:20.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664223 2024-05-15T12:42:20.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612 1-42949673087 2-64424509558 3-335007449132 4-107374182505 5-365072220198 6-438086664223' 2024-05-15T12:42:20.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:20.917 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-15T12:42:21.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724629 2024-05-15T12:42:21.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724629 2024-05-15T12:42:21.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612 1-42949673087 2-64424509558 3-335007449132 4-107374182505 5-365072220198 6-438086664223 7-167503724629' 2024-05-15T12:42:21.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:21.107 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-15T12:42:21.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561099 2024-05-15T12:42:21.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561099 2024-05-15T12:42:21.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612 1-42949673087 2-64424509558 3-335007449132 4-107374182505 5-365072220198 6-438086664223 7-167503724629 8-188978561099' 2024-05-15T12:42:21.276 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:21.276 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-05-15T12:42:21.556 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=472446402586 2024-05-15T12:42:21.556 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 472446402586 2024-05-15T12:42:21.556 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612 1-42949673087 2-64424509558 3-335007449132 4-107374182505 5-365072220198 6-438086664223 7-167503724629 8-188978561099 9-472446402586' 2024-05-15T12:42:21.556 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:42:21.557 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-05-15T12:42:21.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928234042 2024-05-15T12:42:21.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928234042 2024-05-15T12:42:21.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612 1-42949673087 2-64424509558 3-335007449132 4-107374182505 5-365072220198 6-438086664223 7-167503724629 8-188978561099 9-472446402586 10-231928234042' 2024-05-15T12:42:21.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:21.920 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836612 2024-05-15T12:42:21.920 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:21.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T12:42:21.921 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836612 2024-05-15T12:42:21.922 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:21.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836612 2024-05-15T12:42:21.923 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 21474836612 2024-05-15T12:42:21.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836612' 2024-05-15T12:42:21.923 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T12:42:22.345 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836612 -lt 21474836612 2024-05-15T12:42:22.345 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:22.346 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673087 2024-05-15T12:42:22.346 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:22.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T12:42:22.348 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:22.348 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673087 2024-05-15T12:42:22.349 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673087 2024-05-15T12:42:22.350 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 42949673087 2024-05-15T12:42:22.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673087' 2024-05-15T12:42:22.350 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T12:42:22.753 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673087 -lt 42949673087 2024-05-15T12:42:22.753 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:22.755 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509558 2024-05-15T12:42:22.755 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:22.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T12:42:22.757 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509558 2024-05-15T12:42:22.757 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:22.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509558 2024-05-15T12:42:22.759 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 64424509558 2024-05-15T12:42:22.759 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509558' 2024-05-15T12:42:22.759 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T12:42:23.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509559 -lt 64424509558 2024-05-15T12:42:23.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:23.271 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-335007449132 2024-05-15T12:42:23.271 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:23.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T12:42:23.274 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-335007449132 2024-05-15T12:42:23.274 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:23.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449132 2024-05-15T12:42:23.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 335007449132' 2024-05-15T12:42:23.275 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 335007449132 2024-05-15T12:42:23.276 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T12:42:23.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449132 -lt 335007449132 2024-05-15T12:42:23.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:23.706 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182505 2024-05-15T12:42:23.706 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:23.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T12:42:23.707 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182505 2024-05-15T12:42:23.708 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:23.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182505 2024-05-15T12:42:23.710 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.4 seq 107374182505 2024-05-15T12:42:23.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182505' 2024-05-15T12:42:23.710 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T12:42:24.142 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182505 -lt 107374182505 2024-05-15T12:42:24.143 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:24.143 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:24.144 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-365072220198 2024-05-15T12:42:24.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-15T12:42:24.146 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-365072220198 2024-05-15T12:42:24.146 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:24.148 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220198 2024-05-15T12:42:24.148 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.5 seq 365072220198 2024-05-15T12:42:24.148 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 365072220198' 2024-05-15T12:42:24.148 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-15T12:42:24.578 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220199 -lt 365072220198 2024-05-15T12:42:24.578 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:24.579 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-438086664223 2024-05-15T12:42:24.579 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:24.580 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-15T12:42:24.581 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-438086664223 2024-05-15T12:42:24.581 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:24.583 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.6 seq 438086664223 2024-05-15T12:42:24.583 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664223 2024-05-15T12:42:24.583 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 438086664223' 2024-05-15T12:42:24.584 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-15T12:42:25.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664224 -lt 438086664223 2024-05-15T12:42:25.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:25.012 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724629 2024-05-15T12:42:25.012 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:25.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-05-15T12:42:25.015 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724629 2024-05-15T12:42:25.015 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:25.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724629 2024-05-15T12:42:25.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724629' 2024-05-15T12:42:25.017 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.7 seq 167503724629 2024-05-15T12:42:25.017 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-05-15T12:42:25.460 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724630 -lt 167503724629 2024-05-15T12:42:25.460 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:25.461 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561099 2024-05-15T12:42:25.461 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:25.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-15T12:42:25.463 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561099 2024-05-15T12:42:25.463 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:25.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561099 2024-05-15T12:42:25.465 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.8 seq 188978561099 2024-05-15T12:42:25.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561099' 2024-05-15T12:42:25.465 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-15T12:42:25.886 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561099 -lt 188978561099 2024-05-15T12:42:25.886 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:25.887 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-472446402586 2024-05-15T12:42:25.887 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:25.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-05-15T12:42:25.890 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-472446402586 2024-05-15T12:42:25.890 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:25.891 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=472446402586 2024-05-15T12:42:25.891 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 472446402586' 2024-05-15T12:42:25.892 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.9 seq 472446402586 2024-05-15T12:42:25.892 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-05-15T12:42:26.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 472446402586 -lt 472446402586 2024-05-15T12:42:26.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:42:26.320 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-231928234042 2024-05-15T12:42:26.320 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:42:26.322 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-05-15T12:42:26.322 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-231928234042 2024-05-15T12:42:26.322 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:42:26.324 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928234042 2024-05-15T12:42:26.324 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 231928234042' 2024-05-15T12:42:26.325 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.10 seq 231928234042 2024-05-15T12:42:26.325 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-05-15T12:42:26.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928234042 -lt 231928234042 2024-05-15T12:42:26.796 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T12:42:26.796 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:42:26.796 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:42:27.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-05-15T12:42:27.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T12:42:27.389 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T12:42:27.389 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T12:42:27.389 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T12:42:27.389 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T12:42:27.389 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T12:42:27.389 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T12:42:27.832 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-05-15T12:42:27.833 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T12:42:27.833 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:42:27.833 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:42:28.371 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-05-15T12:42:28.371 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T12:42:28.371 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T12:42:28.372 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-05-15T12:42:28.372 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T12:42:28.372 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-15T12:42:28.372 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-05-15T12:42:28.373 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:42:28.885 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-05-15T12:42:28.885 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:42:28.886 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:42:28.886 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T12:42:28.886 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:42:28.886 INFO:tasks.workunit.client.0.smithi005.stderr:10' 2024-05-15T12:42:28.886 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-05-15T12:42:28.886 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: test '5 1 9 4 3 10' = '5 1 9 4 3 10' 2024-05-15T12:42:28.886 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:152: rados_osds_out_in: rm td/test-erasure-code/ORIGINAL 2024-05-15T12:42:28.886 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:229: TEST_rados_put_get_jerasure: delete_pool pool-jerasure 2024-05-15T12:42:28.887 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-jerasure 2024-05-15T12:42:28.887 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-15T12:42:29.528 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-jerasure' does not exist 2024-05-15T12:42:29.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:230: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile rm profile-jerasure 2024-05-15T12:42:31.089 INFO:tasks.workunit.client.0.smithi005.stderr:erasure-code-profile profile-jerasure does not exist 2024-05-15T12:42:31.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-05-15T12:42:31.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_advanced td/test-erasure-code 2024-05-15T12:42:31.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:156: TEST_rados_put_get_lrc_advanced: local dir=td/test-erasure-code 2024-05-15T12:42:31.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:157: TEST_rados_put_get_lrc_advanced: local poolname=pool-lrc-a 2024-05-15T12:42:31.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:158: TEST_rados_put_get_lrc_advanced: local profile=profile-lrc-a 2024-05-15T12:42:31.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:160: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile set profile-lrc-a plugin=lrc mapping=DD_ 'crush-steps=[ [ "chooseleaf", "osd", 0 ] ]' 'layers=[ [ "DDc", "" ] ]' 2024-05-15T12:42:32.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:165: TEST_rados_put_get_lrc_advanced: create_pool pool-lrc-a 12 12 erasure profile-lrc-a 2024-05-15T12:42:32.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-lrc-a 12 12 erasure profile-lrc-a 2024-05-15T12:42:33.032 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-lrc-a' already exists 2024-05-15T12:42:33.051 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T12:42:34.053 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:168: TEST_rados_put_get_lrc_advanced: rados_put_get td/test-erasure-code pool-lrc-a 2024-05-15T12:42:34.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-05-15T12:42:34.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc-a 2024-05-15T12:42:34.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-05-15T12:42:34.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:42:34.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-05-15T12:42:34.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:42:34.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-05-15T12:42:34.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:42:34.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-05-15T12:42:34.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:42:34.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-05-15T12:42:34.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc-a put SOMETHING td/test-erasure-code/ORIGINAL 2024-05-15T12:42:35.508 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2024-05-15T12:42:35.831 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-15T12:42:35.832 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-05-15T12:42:35.837 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc-a SOMETHING 2024-05-15T12:42:35.837 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc-a 2024-05-15T12:42:35.837 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-15T12:42:35.837 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2024-05-15T12:42:35.838 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:42:36.881 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-05-15T12:42:36.881 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:42:36.881 INFO:tasks.workunit.client.0.smithi005.stderr:3' 2024-05-15T12:42:36.881 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 9 3 2024-05-15T12:42:36.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '9' '3') 2024-05-15T12:42:36.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-05-15T12:42:36.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2024-05-15T12:42:36.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 3 2024-05-15T12:42:37.998 INFO:tasks.workunit.client.0.smithi005.stderr:osd.3 is already out. 2024-05-15T12:42:38.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-05-15T12:42:43.017 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc-a SOMETHING 2024-05-15T12:42:43.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc-a 2024-05-15T12:42:43.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-15T12:42:43.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<3\>' 2024-05-15T12:42:43.019 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2024-05-15T12:42:43.019 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:42:43.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-05-15T12:42:43.447 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:42:43.447 INFO:tasks.workunit.client.0.smithi005.stderr:7' 2024-05-15T12:42:43.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 9 7 2024-05-15T12:42:43.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2024-05-15T12:42:43.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-15T12:42:43.516 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 3 2024-05-15T12:42:44.096 INFO:tasks.workunit.client.0.smithi005.stderr:osd.3 is already in. 2024-05-15T12:42:44.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-05-15T12:42:44.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:170: TEST_rados_put_get_lrc_advanced: delete_pool pool-lrc-a 2024-05-15T12:42:44.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-lrc-a 2024-05-15T12:42:44.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-lrc-a pool-lrc-a --yes-i-really-really-mean-it 2024-05-15T12:42:45.773 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-lrc-a' does not exist 2024-05-15T12:42:45.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:171: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile rm profile-lrc-a 2024-05-15T12:42:47.225 INFO:tasks.workunit.client.0.smithi005.stderr:erasure-code-profile profile-lrc-a does not exist 2024-05-15T12:42:47.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-05-15T12:42:47.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_kml td/test-erasure-code 2024-05-15T12:42:47.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:175: TEST_rados_put_get_lrc_kml: local dir=td/test-erasure-code 2024-05-15T12:42:47.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:176: TEST_rados_put_get_lrc_kml: local poolname=pool-lrc 2024-05-15T12:42:47.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:177: TEST_rados_put_get_lrc_kml: local profile=profile-lrc 2024-05-15T12:42:47.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:179: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile set profile-lrc plugin=lrc k=4 m=2 l=3 crush-failure-domain=osd 2024-05-15T12:42:48.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:183: TEST_rados_put_get_lrc_kml: create_pool pool-lrc 12 12 erasure profile-lrc 2024-05-15T12:42:48.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-lrc 12 12 erasure profile-lrc 2024-05-15T12:42:49.248 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-lrc' already exists 2024-05-15T12:42:49.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T12:42:50.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:186: TEST_rados_put_get_lrc_kml: rados_put_get td/test-erasure-code pool-lrc 2024-05-15T12:42:50.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-05-15T12:42:50.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc 2024-05-15T12:42:50.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-05-15T12:42:50.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:42:50.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-05-15T12:42:50.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:42:50.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-05-15T12:42:50.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:42:50.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-05-15T12:42:50.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:42:50.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-05-15T12:42:50.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc put SOMETHING td/test-erasure-code/ORIGINAL 2024-05-15T12:42:51.516 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2024-05-15T12:42:51.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-15T12:42:51.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-05-15T12:42:51.928 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc SOMETHING 2024-05-15T12:42:51.928 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc 2024-05-15T12:42:51.928 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-15T12:42:51.929 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc SOMETHING 2024-05-15T12:42:51.929 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:42:52.469 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=8 2024-05-15T12:42:52.469 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T12:42:52.469 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T12:42:52.469 INFO:tasks.workunit.client.0.smithi005.stderr:10 2024-05-15T12:42:52.469 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T12:42:52.469 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:42:52.469 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:42:52.469 INFO:tasks.workunit.client.0.smithi005.stderr:7' 2024-05-15T12:42:52.469 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 8 6 0 10 5 3 9 7 2024-05-15T12:42:52.469 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('8' '6' '0' '10' '5' '3' '9' '7') 2024-05-15T12:42:52.469 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-05-15T12:42:52.470 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=7 2024-05-15T12:42:52.470 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 7 2024-05-15T12:42:53.309 INFO:tasks.workunit.client.0.smithi005.stderr:osd.7 is already out. 2024-05-15T12:42:53.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-05-15T12:42:58.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc SOMETHING 2024-05-15T12:42:58.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc 2024-05-15T12:42:58.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-15T12:42:58.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<7\>' 2024-05-15T12:42:58.333 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc SOMETHING 2024-05-15T12:42:58.333 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:42:58.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=8 2024-05-15T12:42:58.763 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T12:42:58.763 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T12:42:58.763 INFO:tasks.workunit.client.0.smithi005.stderr:10 2024-05-15T12:42:58.763 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T12:42:58.763 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:42:58.763 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:42:58.763 INFO:tasks.workunit.client.0.smithi005.stderr:4' 2024-05-15T12:42:58.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 8 6 0 10 5 3 9 4 2024-05-15T12:42:58.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2024-05-15T12:42:58.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-15T12:42:58.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 7 2024-05-15T12:43:00.075 INFO:tasks.workunit.client.0.smithi005.stderr:osd.7 is already in. 2024-05-15T12:43:00.097 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-05-15T12:43:00.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:188: TEST_rados_put_get_lrc_kml: delete_pool pool-lrc 2024-05-15T12:43:00.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-lrc 2024-05-15T12:43:00.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-lrc pool-lrc --yes-i-really-really-mean-it 2024-05-15T12:43:02.077 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-lrc' does not exist 2024-05-15T12:43:02.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:189: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile rm profile-lrc 2024-05-15T12:43:04.061 INFO:tasks.workunit.client.0.smithi005.stderr:erasure-code-profile profile-lrc does not exist 2024-05-15T12:43:04.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-05-15T12:43:04.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_shec td/test-erasure-code 2024-05-15T12:43:04.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:234: TEST_rados_put_get_shec: local dir=td/test-erasure-code 2024-05-15T12:43:04.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:236: TEST_rados_put_get_shec: local poolname=pool-shec 2024-05-15T12:43:04.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:237: TEST_rados_put_get_shec: local profile=profile-shec 2024-05-15T12:43:04.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:239: TEST_rados_put_get_shec: ceph osd erasure-code-profile set profile-shec plugin=shec k=2 m=1 c=1 crush-failure-domain=osd 2024-05-15T12:43:05.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:243: TEST_rados_put_get_shec: create_pool pool-shec 12 12 erasure profile-shec 2024-05-15T12:43:05.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-shec 12 12 erasure profile-shec 2024-05-15T12:43:06.124 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-shec' already exists 2024-05-15T12:43:06.142 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T12:43:07.145 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:246: TEST_rados_put_get_shec: rados_put_get td/test-erasure-code pool-shec 2024-05-15T12:43:07.145 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-05-15T12:43:07.145 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-shec 2024-05-15T12:43:07.145 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-05-15T12:43:07.145 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:43:07.145 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-05-15T12:43:07.145 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:43:07.145 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-05-15T12:43:07.145 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:43:07.145 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-05-15T12:43:07.146 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-15T12:43:07.146 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-05-15T12:43:07.146 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-shec put SOMETHING td/test-erasure-code/ORIGINAL 2024-05-15T12:43:09.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2024-05-15T12:43:09.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-15T12:43:09.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-05-15T12:43:09.677 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-shec SOMETHING 2024-05-15T12:43:09.677 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-shec 2024-05-15T12:43:09.677 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-15T12:43:09.678 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-shec SOMETHING 2024-05-15T12:43:09.678 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:43:10.353 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=7 2024-05-15T12:43:10.353 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:43:10.353 INFO:tasks.workunit.client.0.smithi005.stderr:1' 2024-05-15T12:43:10.353 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 7 9 1 2024-05-15T12:43:10.353 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('7' '9' '1') 2024-05-15T12:43:10.353 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-05-15T12:43:10.353 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2024-05-15T12:43:10.353 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 1 2024-05-15T12:43:10.958 INFO:tasks.workunit.client.0.smithi005.stderr:osd.1 is already out. 2024-05-15T12:43:10.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-05-15T12:43:15.978 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-shec SOMETHING 2024-05-15T12:43:15.978 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-shec 2024-05-15T12:43:15.978 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-15T12:43:15.978 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<1\>' 2024-05-15T12:43:15.979 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-shec SOMETHING 2024-05-15T12:43:15.979 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:43:16.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=7 2024-05-15T12:43:16.463 INFO:tasks.workunit.client.0.smithi005.stderr:9 2024-05-15T12:43:16.463 INFO:tasks.workunit.client.0.smithi005.stderr:6' 2024-05-15T12:43:16.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 7 9 6 2024-05-15T12:43:16.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2024-05-15T12:43:16.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-15T12:43:16.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 1 2024-05-15T12:43:17.046 INFO:tasks.workunit.client.0.smithi005.stderr:osd.1 is already in. 2024-05-15T12:43:17.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-05-15T12:43:17.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:248: TEST_rados_put_get_shec: delete_pool pool-shec 2024-05-15T12:43:17.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-shec 2024-05-15T12:43:17.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-shec pool-shec --yes-i-really-really-mean-it 2024-05-15T12:43:18.771 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-shec' does not exist 2024-05-15T12:43:18.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:249: TEST_rados_put_get_shec: ceph osd erasure-code-profile rm profile-shec 2024-05-15T12:43:20.671 INFO:tasks.workunit.client.0.smithi005.stderr:erasure-code-profile profile-shec does not exist 2024-05-15T12:43:20.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:51: run: delete_pool ecpool 2024-05-15T12:43:20.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=ecpool 2024-05-15T12:43:20.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete ecpool ecpool --yes-i-really-really-mean-it 2024-05-15T12:43:21.513 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'ecpool' does not exist 2024-05-15T12:43:21.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:52: run: teardown td/test-erasure-code 2024-05-15T12:43:21.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-05-15T12:43:21.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:43:21.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-05-15T12:43:21.531 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:43:21.532 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:43:21.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:43:21.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:43:21.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:43:21.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:43:21.851 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:43:21.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:43:21.852 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:43:21.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:43:21.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:43:21.854 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:43:21.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:43:21.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:43:21.856 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:43:21.856 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:43:21.857 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:43:21.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:43:21.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-05-15T12:43:21.971 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:43:21.972 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:43:21.972 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:43:21.972 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48542 2024-05-15T12:43:21.974 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:43:21.974 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:43:21.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-05-15T12:43:21.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/test-erasure-code 0 2024-05-15T12:43:21.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-05-15T12:43:21.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-05-15T12:43:21.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-05-15T12:43:21.975 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:43:21.975 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:43:21.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:43:21.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:43:21.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:43:21.977 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:43:21.977 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:43:21.978 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:43:21.979 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:43:21.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:43:21.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:43:21.980 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:43:21.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:43:21.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:43:21.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:43:21.982 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:43:21.983 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:43:21.984 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-05-15T12:43:21.984 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-05-15T12:43:21.985 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:43:21.986 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:43:21.986 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48542 2024-05-15T12:43:21.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48542 2024-05-15T12:43:21.988 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:43:21.988 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:43:21.988 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-05-15T12:43:21.988 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-15T12:43:21.988 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-15T12:43:22.042 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-eio.sh... 2024-05-15T12:43:22.042 DEBUG:teuthology.orchestra.run.smithi005:workunit test erasure-code/test-erasure-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae2481253f9b2fa3451954e615eedb67ebdd5314 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh 2024-05-15T12:43:22.097 INFO:tasks.workunit.client.0.smithi005.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-05-15T12:43:22.104 INFO:tasks.workunit.client.0.smithi005.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-05-15T12:43:22.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-05-15T12:43:22.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-05-15T12:43:22.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-05-15T12:43:22.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-05-15T12:43:22.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-05-15T12:43:22.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-05-15T12:43:22.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-05-15T12:43:22.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-05-15T12:43:22.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/test-erasure-eio 2024-05-15T12:43:22.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:22: run: local dir=td/test-erasure-eio 2024-05-15T12:43:22.105 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:23: run: shift 2024-05-15T12:43:22.105 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: export CEPH_MON=127.0.0.1:7112 2024-05-15T12:43:22.105 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: CEPH_MON=127.0.0.1:7112 2024-05-15T12:43:22.105 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:26: run: export CEPH_ARGS 2024-05-15T12:43:22.105 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: uuidgen 2024-05-15T12:43:22.106 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: CEPH_ARGS+='--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none ' 2024-05-15T12:43:22.106 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7112 ' 2024-05-15T12:43:22.106 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2024-05-15T12:43:22.106 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: set 2024-05-15T12:43:22.106 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-05-15T12:43:22.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: local 'funcs=TEST_ec_backfill_unfound 2024-05-15T12:43:22.108 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_ec_object_attr_read_error 2024-05-15T12:43:22.108 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_ec_recovery_multiple_errors 2024-05-15T12:43:22.108 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_ec_recovery_multiple_objects 2024-05-15T12:43:22.109 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_ec_recovery_multiple_objects_eio 2024-05-15T12:43:22.109 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_ec_recovery_unfound 2024-05-15T12:43:22.109 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_ec_single_recovery_error 2024-05-15T12:43:22.109 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_rados_get_bad_size_shard_0 2024-05-15T12:43:22.109 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_rados_get_bad_size_shard_1 2024-05-15T12:43:22.109 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_rados_get_subread_eio_shard_0 2024-05-15T12:43:22.109 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_rados_get_subread_eio_shard_1 2024-05-15T12:43:22.109 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_rados_get_subread_missing 2024-05-15T12:43:22.109 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_rados_get_with_subreadall_eio_shard_0 2024-05-15T12:43:22.109 INFO:tasks.workunit.client.0.smithi005.stderr:TEST_rados_get_with_subreadall_eio_shard_1' 2024-05-15T12:43:22.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-15T12:43:22.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-15T12:43:22.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-15T12:43:22.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-15T12:43:22.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-15T12:43:22.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:43:22.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-15T12:43:22.110 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:43:22.110 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:43:22.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:43:22.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:43:22.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:43:22.111 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:43:22.111 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:43:22.111 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:43:22.111 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:43:22.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:43:22.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:43:22.113 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:43:22.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:43:22.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:43:22.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:43:22.115 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:43:22.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:43:22.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:43:22.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-15T12:43:22.118 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:43:22.118 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:43:22.118 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:43:22.118 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80702 2024-05-15T12:43:22.119 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:43:22.119 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:43:22.119 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-15T12:43:22.121 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T12:43:22.121 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:43:22.121 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:43:22.121 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.80702 2024-05-15T12:43:22.122 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T12:43:22.123 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-05-15T12:43:22.123 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-05-15T12:43:22.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T12:43:22.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-15T12:43:22.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-15T12:43:22.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-15T12:43:22.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T12:43:22.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T12:43:22.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T12:43:22.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-15T12:43:22.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-15T12:43:22.517 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T12:43:22.517 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:43:22.517 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:43:22.517 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:43:22.517 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:43:22.517 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:43:22.517 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:43:22.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-15T12:43:22.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T12:43:22.763 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T12:43:22.763 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:43:22.763 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:43:22.763 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T12:43:22.763 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T12:43:22.763 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:43:22.763 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:43:22.763 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:43:22.764 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:43:22.764 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:43:22.764 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:43:22.764 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T12:43:22.764 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:43:22.765 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.80702/ceph-mon.a.asok config get fsid 2024-05-15T12:43:22.882 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T12:43:22.882 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:43:22.882 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:43:22.882 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T12:43:22.883 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T12:43:22.883 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:43:22.883 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:43:22.883 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:43:22.883 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:43:22.883 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:43:22.883 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:43:22.884 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T12:43:22.884 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:43:22.884 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.80702/ceph-mon.a.asok config get mon_host 2024-05-15T12:43:23.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-15T12:43:23.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-15T12:43:23.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T12:43:23.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T12:43:23.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T12:43:23.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-15T12:43:23.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T12:43:23.917 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T12:43:23.917 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:43:23.917 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:43:23.918 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:43:23.918 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:43:23.918 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:43:23.918 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:43:23.918 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:43:23.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:43:23.992 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-15T12:43:23.992 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-15T12:43:24.340 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd' already exists 2024-05-15T12:43:24.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T12:43:25.359 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-15T12:43:25.359 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:43:25.359 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:43:25.360 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:43:25.360 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:43:25.361 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:43:25.361 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T12:43:25.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-15T12:43:25.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-mon.a.asok log flush 2024-05-15T12:43:25.461 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:43:25.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-15T12:43:25.475 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:43:22.572+0000 7f61232b6c80 0 load: jerasure load: lrc 2024-05-15T12:43:25.476 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_backfill_unfound td/test-erasure-eio 2024-05-15T12:43:25.476 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:511: TEST_ec_backfill_unfound: local dir=td/test-erasure-eio 2024-05-15T12:43:25.476 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:512: TEST_ec_backfill_unfound: local objname=myobject 2024-05-15T12:43:25.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:513: TEST_ec_backfill_unfound: local lastobj=300 2024-05-15T12:43:25.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:515: TEST_ec_backfill_unfound: local testobj=obj250 2024-05-15T12:43:25.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:517: TEST_ec_backfill_unfound: ORIG_ARGS='--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:43:25.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:518: TEST_ec_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-15T12:43:25.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:519: TEST_ec_backfill_unfound: setup_osds 5 2024-05-15T12:43:25.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2024-05-15T12:43:25.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-15T12:43:25.477 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2024-05-15T12:43:25.478 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2024-05-15T12:43:25.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:43:25.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-15T12:43:25.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:43:25.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:43:25.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T12:43:25.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:43:25.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-15T12:43:25.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-15T12:43:25.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:43:25.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:43:25.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:43:25.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-15T12:43:25.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-15T12:43:25.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:43:25.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:43:25.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:43:25.484 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:43:25.484 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:43:25.484 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:43:25.484 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:43:25.484 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:43:25.484 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:43:25.484 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:43:25.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:43:25.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:43:25.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:43:25.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:43:25.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:43:25.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:43:25.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:43:25.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:43:25.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:43:25.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:43:25.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:43:25.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:43:25.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-15T12:43:25.485 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:43:25.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c426aa58-ceb0-4aba-a91b-a9dc65de8705 2024-05-15T12:43:25.486 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 c426aa58-ceb0-4aba-a91b-a9dc65de8705 2024-05-15T12:43:25.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c426aa58-ceb0-4aba-a91b-a9dc65de8705' 2024-05-15T12:43:25.487 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:43:25.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDtrURmv6BUHxAAEThuId7ZcCKfALo21q87XQ== 2024-05-15T12:43:25.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDtrURmv6BUHxAAEThuId7ZcCKfALo21q87XQ=="}' 2024-05-15T12:43:25.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c426aa58-ceb0-4aba-a91b-a9dc65de8705 -i td/test-erasure-eio/0/new.json 2024-05-15T12:43:25.841 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:43:25.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-15T12:43:25.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDtrURmv6BUHxAAEThuId7ZcCKfALo21q87XQ== --osd-uuid c426aa58-ceb0-4aba-a91b-a9dc65de8705 2024-05-15T12:43:25.899 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:43:25.912+0000 7f77d15b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:43:25.926 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:43:25.940+0000 7f77d15b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:43:25.928 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:43:25.940+0000 7f77d15b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:43:25.929 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:43:25.940+0000 7f77d15b37c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T12:43:25.929 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:43:25.940+0000 7f77d15b37c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T12:43:25.929 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:43:25.940+0000 7f77d15b37c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T12:43:25.958 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:43:25.972+0000 7f77d15b37c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-15T12:43:33.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-15T12:43:33.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:43:33.015 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T12:43:33.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T12:43:33.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:43:33.581 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T12:43:33.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T12:43:33.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:43:33.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:43:33.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:43:33.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:43:33.619 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:43:33.632+0000 7f7d14e577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:43:33.640 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:43:33.652+0000 7f7d14e577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:43:33.642 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:43:33.656+0000 7f7d14e577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:43:34.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T12:43:34.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:43:34.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T12:43:34.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:43:34.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:43:34.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:43:34.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:43:34.007 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:43:34.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:43:34.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:43:34.204 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:43:34.216+0000 7f7d14e577c0 -1 Falling back to public interface 2024-05-15T12:43:34.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:43:35.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:43:35.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:43:35.446 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:43:35.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:43:35.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:43:35.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:43:35.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:43:36.805 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:43:36.815+0000 7f7d14e577c0 -1 osd.0 0 log_to_monitors true 2024-05-15T12:43:36.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:43:36.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:43:36.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:43:36.877 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:43:36.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:43:36.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:43:37.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:43:38.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:43:38.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:43:38.520 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:43:38.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:43:38.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:43:38.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:43:38.984 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:43:39.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:43:39.987 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:43:39.987 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:43:39.987 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:43:39.987 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:43:39.987 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:43:40.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:43:41.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:43:41.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:43:41.448 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:43:41.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:43:41.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:43:41.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:43:41.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:43:42.977 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:43:42.978 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:43:42.978 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:43:42.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:43:42.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:43:42.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:43:43.437 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:43:44.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:43:44.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:43:44.440 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:43:44.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:43:44.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:43:44.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:43:44.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:43:45.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:43:45.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:43:45.998 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:43:45.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:43:45.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:43:45.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:43:46.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:43:47.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:43:47.466 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:43:47.466 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:43:47.466 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:43:47.466 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:43:47.467 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:43:48.036 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:43:49.038 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:43:49.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:43:49.039 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:43:49.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:43:49.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:43:49.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:43:49.524 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:43:50.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:43:50.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:43:50.526 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:43:50.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:43:50.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:43:50.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:43:51.498 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:43:52.500 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:43:52.500 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:43:52.501 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:43:52.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:43:52.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:43:52.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:43:53.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:43:54.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:43:54.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:43:54.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:43:54.260 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:43:54.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:43:54.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:43:55.209 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:43:56.211 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:43:56.211 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:43:56.211 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:43:56.211 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:43:56.212 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:43:56.212 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:43:57.049 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:43:58.051 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:43:58.051 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:43:58.051 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:43:58.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:43:58.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:43:58.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:43:58.806 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:43:59.807 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:43:59.807 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:43:59.808 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:43:59.808 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:43:59.808 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:43:59.808 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:44:00.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:01.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:01.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:01.920 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:44:01.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:44:01.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:01.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:44:02.528 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3760732104,v1:127.0.0.1:6803/3760732104] [v2:127.0.0.1:6804/3760732104,v1:127.0.0.1:6805/3760732104] exists,up c426aa58-ceb0-4aba-a91b-a9dc65de8705 2024-05-15T12:44:02.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:44:02.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:44:02.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:44:02.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:44:02.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-15T12:44:02.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:44:02.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:44:02.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-15T12:44:02.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:44:02.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-15T12:44:02.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-15T12:44:02.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:44:02.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:44:02.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:44:02.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-15T12:44:02.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-15T12:44:02.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:44:02.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:44:02.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:44:02.533 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:44:02.533 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:44:02.533 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:44:02.533 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:44:02.533 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:44:02.533 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:44:02.533 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:44:02.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:44:02.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:44:02.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:44:02.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:44:02.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:44:02.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:44:02.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:44:02.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:44:02.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:44:02.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:44:02.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:44:02.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:44:02.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-15T12:44:02.535 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:44:02.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8c893ac4-9ca5-4523-b63b-e811ab72abcf 2024-05-15T12:44:02.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8c893ac4-9ca5-4523-b63b-e811ab72abcf' 2024-05-15T12:44:02.535 INFO:tasks.workunit.client.0.smithi005.stdout:add osd1 8c893ac4-9ca5-4523-b63b-e811ab72abcf 2024-05-15T12:44:02.536 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:44:02.562 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQASrkRmemdFIhAAzMAmMJWbKkEIl/A9Bh+y4A== 2024-05-15T12:44:02.562 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQASrkRmemdFIhAAzMAmMJWbKkEIl/A9Bh+y4A=="}' 2024-05-15T12:44:02.562 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8c893ac4-9ca5-4523-b63b-e811ab72abcf -i td/test-erasure-eio/1/new.json 2024-05-15T12:44:03.569 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:44:03.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-15T12:44:03.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQASrkRmemdFIhAAzMAmMJWbKkEIl/A9Bh+y4A== --osd-uuid 8c893ac4-9ca5-4523-b63b-e811ab72abcf 2024-05-15T12:44:03.629 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:03.639+0000 7f11654c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:44:03.650 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:03.663+0000 7f11654c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:44:03.651 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:03.663+0000 7f11654c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:44:03.651 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:03.663+0000 7f11654c17c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T12:44:03.651 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:03.663+0000 7f11654c17c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T12:44:03.652 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:03.663+0000 7f11654c17c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T12:44:03.741 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:03.751+0000 7f11654c17c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-15T12:44:10.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-15T12:44:10.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:44:10.777 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd1 key to auth repository 2024-05-15T12:44:10.777 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-15T12:44:10.777 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:44:11.361 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.1 2024-05-15T12:44:11.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-15T12:44:11.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:44:11.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:44:11.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:44:11.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:44:11.398 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:11.411+0000 7f097d6407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:44:11.440 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:11.451+0000 7f097d6407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:44:11.442 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:11.455+0000 7f097d6407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:44:11.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-15T12:44:11.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:44:11.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-15T12:44:11.822 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:44:11.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:44:11.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:44:11.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:11.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:44:11.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:11.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:44:12.007 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:12.019+0000 7f097d6407c0 -1 Falling back to public interface 2024-05-15T12:44:12.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:13.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:13.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:13.282 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:44:13.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:44:13.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:13.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:44:13.726 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:14.685 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:14.695+0000 7f097d6407c0 -1 osd.1 0 log_to_monitors true 2024-05-15T12:44:14.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:14.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:14.727 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:44:14.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:44:14.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:14.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:44:15.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:16.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:16.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:16.489 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:44:16.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:44:16.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:16.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:44:16.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:17.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:17.926 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:44:17.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:17.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:44:17.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:17.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:44:18.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:19.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:19.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:19.365 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:44:19.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:44:19.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:19.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:44:19.942 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:20.943 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:20.944 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:44:20.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:20.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:44:20.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:20.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:44:21.469 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:22.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:22.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:22.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:44:22.472 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:44:22.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:22.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:44:22.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:23.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:23.919 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:44:23.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:23.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:44:23.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:23.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:44:24.374 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:25.376 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:25.376 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:25.377 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:44:25.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:44:25.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:25.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:44:25.946 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:26.948 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:26.948 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:26.948 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:44:26.949 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:44:26.949 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:26.949 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:44:27.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:28.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:28.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:28.503 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:44:28.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:44:28.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:28.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:44:29.096 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:30.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:30.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:30.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:44:30.099 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:44:30.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:30.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:44:30.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:31.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:31.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:31.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:44:31.748 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:44:31.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:31.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:44:32.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:33.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:33.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:33.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:44:33.872 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:44:33.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:33.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:44:34.698 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:35.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:35.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:35.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:44:35.700 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:44:35.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:35.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:44:36.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:37.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:37.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:37.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:44:37.710 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:44:37.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:37.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:44:38.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:39.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:39.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:39.449 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:44:39.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:44:39.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:39.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:44:40.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:41.371 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:41.372 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:41.372 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T12:44:41.372 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T12:44:41.373 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:41.373 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:44:42.034 INFO:tasks.workunit.client.0.smithi005.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3123279250,v1:127.0.0.1:6811/3123279250] [v2:127.0.0.1:6812/3123279250,v1:127.0.0.1:6813/3123279250] exists,up 8c893ac4-9ca5-4523-b63b-e811ab72abcf 2024-05-15T12:44:42.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:44:42.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:44:42.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:44:42.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:44:42.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-15T12:44:42.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:44:42.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:44:42.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-15T12:44:42.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:44:42.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-15T12:44:42.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-15T12:44:42.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:44:42.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:44:42.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:44:42.038 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-15T12:44:42.038 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-15T12:44:42.038 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:44:42.038 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:44:42.038 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:44:42.038 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:44:42.038 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:44:42.038 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:44:42.039 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:44:42.039 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:44:42.039 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:44:42.039 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:44:42.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:44:42.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:44:42.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:44:42.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:44:42.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:44:42.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:44:42.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:44:42.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:44:42.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:44:42.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:44:42.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:44:42.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:44:42.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-15T12:44:42.040 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:44:42.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6a695c3c-f227-4356-a06e-53f491f74662 2024-05-15T12:44:42.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6a695c3c-f227-4356-a06e-53f491f74662' 2024-05-15T12:44:42.042 INFO:tasks.workunit.client.0.smithi005.stdout:add osd2 6a695c3c-f227-4356-a06e-53f491f74662 2024-05-15T12:44:42.043 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:44:42.069 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA6rkRm29HtBBAAAcobcjTqtwmQ7E491J/7xA== 2024-05-15T12:44:42.069 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA6rkRm29HtBBAAAcobcjTqtwmQ7E491J/7xA=="}' 2024-05-15T12:44:42.069 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6a695c3c-f227-4356-a06e-53f491f74662 -i td/test-erasure-eio/2/new.json 2024-05-15T12:44:42.668 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:44:42.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-15T12:44:42.687 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA6rkRm29HtBBAAAcobcjTqtwmQ7E491J/7xA== --osd-uuid 6a695c3c-f227-4356-a06e-53f491f74662 2024-05-15T12:44:42.719 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:42.730+0000 7f6ddaec97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:44:42.745 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:42.754+0000 7f6ddaec97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:44:42.746 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:42.758+0000 7f6ddaec97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:44:42.747 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:42.758+0000 7f6ddaec97c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T12:44:42.747 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:42.758+0000 7f6ddaec97c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T12:44:42.747 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:42.758+0000 7f6ddaec97c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T12:44:42.785 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:42.794+0000 7f6ddaec97c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-15T12:44:49.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-15T12:44:49.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:44:49.550 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd2 key to auth repository 2024-05-15T12:44:49.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-15T12:44:49.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:44:50.200 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.2 2024-05-15T12:44:50.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-15T12:44:50.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:44:50.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:44:50.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:44:50.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:44:50.238 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:50.250+0000 7fdcd18697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:44:50.268 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:50.278+0000 7fdcd18697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:44:50.269 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:50.278+0000 7fdcd18697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:44:50.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-15T12:44:50.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:44:50.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-15T12:44:50.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:44:50.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:44:50.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:50.641 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:44:50.642 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:44:50.642 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:44:50.642 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:50.840 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:50.850+0000 7fdcd18697c0 -1 Falling back to public interface 2024-05-15T12:44:51.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:52.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:52.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:52.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:44:52.087 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:44:52.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:52.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:44:52.508 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:53.509 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:53.509 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:53.510 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:44:53.510 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:44:53.510 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:53.510 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:44:53.548 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:44:53.558+0000 7fdcd18697c0 -1 osd.2 0 log_to_monitors true 2024-05-15T12:44:54.122 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:55.123 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:55.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:55.124 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:44:55.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:44:55.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:55.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:44:55.678 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:56.679 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:56.679 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:56.679 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:44:56.680 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:44:56.680 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:56.680 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:44:57.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:58.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:58.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:58.202 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:44:58.202 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:44:58.202 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:58.202 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:44:58.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:44:59.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:44:59.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:44:59.762 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:44:59.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:44:59.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:44:59.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:45:00.240 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:01.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:01.242 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:45:01.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:01.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:45:01.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:01.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:45:01.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:02.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:02.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:02.708 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:45:02.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:45:02.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:02.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:45:03.442 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:04.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:04.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:04.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:45:04.444 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:45:04.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:04.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:45:04.891 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:05.893 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:05.893 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:05.893 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:45:05.893 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:45:05.894 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:05.894 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:45:06.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:07.467 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:07.467 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:07.467 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:45:07.467 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:45:07.468 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:07.468 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:45:08.148 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:09.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:09.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:09.151 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:45:09.151 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:45:09.151 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:09.151 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:45:09.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:10.997 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:10.997 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:10.998 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:45:10.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:45:10.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:10.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:45:12.060 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:13.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:13.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:13.062 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:45:13.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:45:13.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:13.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:45:14.044 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:15.046 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:15.046 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:15.046 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:45:15.047 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:45:15.047 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:15.047 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:45:16.460 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:17.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:17.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:17.463 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:45:17.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:45:17.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:17.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:45:18.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:19.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:19.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:19.285 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:45:19.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:45:19.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:19.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:45:19.788 INFO:tasks.workunit.client.0.smithi005.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3875573492,v1:127.0.0.1:6819/3875573492] [v2:127.0.0.1:6820/3875573492,v1:127.0.0.1:6821/3875573492] exists,up 6a695c3c-f227-4356-a06e-53f491f74662 2024-05-15T12:45:19.793 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:45:19.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:45:19.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:45:19.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:45:19.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-15T12:45:19.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:45:19.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:45:19.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-15T12:45:19.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:45:19.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-15T12:45:19.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-15T12:45:19.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:45:19.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:45:19.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:45:19.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-15T12:45:19.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-15T12:45:19.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:45:19.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:45:19.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:45:19.795 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:45:19.795 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:45:19.795 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:45:19.795 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:45:19.796 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:45:19.796 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:45:19.796 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:45:19.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:45:19.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:45:19.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:45:19.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:45:19.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:45:19.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:45:19.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:45:19.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:45:19.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:45:19.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:45:19.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:45:19.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:45:19.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-15T12:45:19.797 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:45:19.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a531aa68-363e-4883-9e7c-e746afaf24a8 2024-05-15T12:45:19.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 a531aa68-363e-4883-9e7c-e746afaf24a8' 2024-05-15T12:45:19.797 INFO:tasks.workunit.client.0.smithi005.stdout:add osd3 a531aa68-363e-4883-9e7c-e746afaf24a8 2024-05-15T12:45:19.798 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:45:19.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBfrkRmGr62MRAAAMvfuQ624trGDlocZHHFXw== 2024-05-15T12:45:19.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBfrkRmGr62MRAAAMvfuQ624trGDlocZHHFXw=="}' 2024-05-15T12:45:19.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a531aa68-363e-4883-9e7c-e746afaf24a8 -i td/test-erasure-eio/3/new.json 2024-05-15T12:45:20.324 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:45:20.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-15T12:45:20.341 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBfrkRmGr62MRAAAMvfuQ624trGDlocZHHFXw== --osd-uuid a531aa68-363e-4883-9e7c-e746afaf24a8 2024-05-15T12:45:20.381 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:45:20.389+0000 7fdb141fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:45:20.383 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:45:20.393+0000 7fdb141fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:45:20.385 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:45:20.393+0000 7fdb141fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:45:20.385 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:45:20.393+0000 7fdb141fc7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T12:45:20.385 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:45:20.393+0000 7fdb141fc7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T12:45:20.385 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:45:20.393+0000 7fdb141fc7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T12:45:20.508 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:45:20.517+0000 7fdb141fc7c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-15T12:45:28.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-15T12:45:28.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:45:28.489 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd3 key to auth repository 2024-05-15T12:45:28.490 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-15T12:45:28.490 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:45:29.098 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.3 2024-05-15T12:45:29.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-15T12:45:29.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:45:29.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:45:29.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:45:29.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:45:29.135 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:45:29.145+0000 7fa588fee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:45:29.179 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:45:29.189+0000 7fa588fee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:45:29.180 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:45:29.189+0000 7fa588fee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:45:29.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-15T12:45:29.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:45:29.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-15T12:45:29.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:45:29.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:45:29.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:29.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:45:29.571 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:45:29.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:29.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:45:29.743 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:45:29.753+0000 7fa588fee7c0 -1 Falling back to public interface 2024-05-15T12:45:29.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:31.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:31.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:31.000 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:45:31.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:45:31.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:31.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:45:31.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:32.379 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:45:32.389+0000 7fa588fee7c0 -1 osd.3 0 log_to_monitors true 2024-05-15T12:45:32.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:32.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:32.503 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:45:32.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:45:32.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:32.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:45:33.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:34.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:34.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:34.311 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:45:34.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:45:34.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:34.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:45:34.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:35.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:35.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:35.861 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:45:35.861 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:45:35.861 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:35.861 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:45:36.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:37.407 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:37.407 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:37.407 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:45:37.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:45:37.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:37.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:45:37.897 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:38.898 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:38.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:38.899 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:45:38.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:45:38.900 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:38.900 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:45:39.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:40.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:40.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:40.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:45:40.411 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:45:40.412 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:40.412 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:45:40.842 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:41.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:41.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:41.844 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:45:41.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:45:41.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:41.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:45:42.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:43.368 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:43.368 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:43.368 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:45:43.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:45:43.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:43.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:45:43.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:44.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:44.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:44.828 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:45:44.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:45:44.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:44.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:45:45.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:46.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:46.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:46.410 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:45:46.410 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:45:46.410 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:46.410 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:45:46.898 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:47.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:47.900 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:47.900 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:45:47.900 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:45:47.900 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:47.900 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:45:48.516 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:49.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:49.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:49.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:45:49.519 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:45:49.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:49.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:45:50.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:51.410 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:51.410 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:51.410 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:45:51.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:45:51.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:51.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:45:52.567 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:53.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:53.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:53.568 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:45:53.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:45:53.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:53.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:45:54.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:55.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:55.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:55.077 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:45:55.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:45:55.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:55.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:45:55.737 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:56.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:56.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:56.739 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:45:56.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:45:56.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:56.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:45:57.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:45:58.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:45:58.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:45:58.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T12:45:58.568 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T12:45:58.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:45:58.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:45:59.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:46:00.028 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:46:00.028 INFO:tasks.workunit.client.0.smithi005.stdout:19 2024-05-15T12:46:00.028 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:00.028 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-15T12:46:00.028 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:00.029 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:46:00.664 INFO:tasks.workunit.client.0.smithi005.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2693852748,v1:127.0.0.1:6827/2693852748] [v2:127.0.0.1:6828/2693852748,v1:127.0.0.1:6829/2693852748] exists,up a531aa68-363e-4883-9e7c-e746afaf24a8 2024-05-15T12:46:00.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:46:00.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:46:00.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:46:00.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:46:00.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-05-15T12:46:00.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:46:00.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:46:00.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-05-15T12:46:00.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:46:00.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-05-15T12:46:00.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-15T12:46:00.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:46:00.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:46:00.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:46:00.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-05-15T12:46:00.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-05-15T12:46:00.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:46:00.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:46:00.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:46:00.667 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:46:00.667 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:46:00.667 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:46:00.668 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:46:00.668 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:46:00.668 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:46:00.668 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:46:00.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:46:00.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:46:00.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:46:00.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:46:00.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:46:00.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:46:00.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:46:00.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:46:00.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:46:00.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:46:00.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:46:00.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:46:00.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-05-15T12:46:00.670 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:46:00.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3f468dd3-1b89-47be-bdd3-b69fc55b54b8 2024-05-15T12:46:00.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 3f468dd3-1b89-47be-bdd3-b69fc55b54b8' 2024-05-15T12:46:00.670 INFO:tasks.workunit.client.0.smithi005.stdout:add osd4 3f468dd3-1b89-47be-bdd3-b69fc55b54b8 2024-05-15T12:46:00.670 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:46:00.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCIrkRm7J5HKhAASci+uw0EW93mZUM5qAzyEQ== 2024-05-15T12:46:00.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCIrkRm7J5HKhAASci+uw0EW93mZUM5qAzyEQ=="}' 2024-05-15T12:46:00.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3f468dd3-1b89-47be-bdd3-b69fc55b54b8 -i td/test-erasure-eio/4/new.json 2024-05-15T12:46:01.466 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:46:01.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-05-15T12:46:01.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCIrkRm7J5HKhAASci+uw0EW93mZUM5qAzyEQ== --osd-uuid 3f468dd3-1b89-47be-bdd3-b69fc55b54b8 2024-05-15T12:46:01.530 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:46:01.541+0000 7fe90943c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:46:01.676 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:46:01.685+0000 7fe90943c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:46:01.678 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:46:01.689+0000 7fe90943c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:46:01.678 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:46:01.689+0000 7fe90943c7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T12:46:01.678 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:46:01.689+0000 7fe90943c7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T12:46:01.678 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:46:01.689+0000 7fe90943c7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T12:46:01.799 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:46:01.813+0000 7fe90943c7c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-05-15T12:46:08.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-05-15T12:46:08.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:46:08.674 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd4 key to auth repository 2024-05-15T12:46:08.674 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-05-15T12:46:08.674 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:46:09.248 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.4 2024-05-15T12:46:09.248 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-05-15T12:46:09.248 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:46:09.248 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:46:09.249 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:46:09.249 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:46:09.286 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:46:09.297+0000 7fae5de617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:46:09.316 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:46:09.325+0000 7fae5de617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:46:09.317 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:46:09.329+0000 7fae5de617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:46:09.682 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-05-15T12:46:09.683 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:46:09.683 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-05-15T12:46:09.683 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:46:09.683 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:46:09.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:46:09.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:09.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:46:09.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:09.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:46:09.875 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:46:09.885+0000 7fae5de617c0 -1 Falling back to public interface 2024-05-15T12:46:10.136 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:46:11.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:46:11.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:11.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:46:11.139 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:46:11.139 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:11.139 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:46:11.567 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:46:12.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:46:12.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:12.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:46:12.569 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:46:12.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:12.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:46:12.629 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:46:12.637+0000 7fae5de617c0 -1 osd.4 0 log_to_monitors true 2024-05-15T12:46:13.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:46:14.216 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:46:14.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:14.217 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:46:14.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:46:14.218 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:14.218 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:46:14.914 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:46:15.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:46:15.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:15.917 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:46:15.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:46:15.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:15.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:46:16.562 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:46:17.563 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:46:17.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:17.564 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:46:17.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:46:17.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:17.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:46:18.122 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:46:19.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:46:19.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:19.124 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:46:19.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:46:19.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:19.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:46:19.552 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:46:20.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:46:20.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:20.555 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:46:20.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:46:20.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:20.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:46:21.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:46:22.065 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:46:22.065 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:22.065 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:46:22.065 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:46:22.066 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:22.066 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:46:22.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:46:23.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:46:23.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:23.603 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:46:23.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:46:23.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:23.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:46:24.078 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:46:25.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:46:25.080 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:46:25.080 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:25.080 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:46:25.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:25.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:46:25.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:46:26.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:46:26.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:26.522 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:46:26.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:46:26.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:26.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:46:27.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:46:28.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:46:28.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:28.054 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:46:28.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:46:28.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:28.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:46:28.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:46:29.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:46:29.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:29.538 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:46:29.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:46:29.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:29.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:46:30.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:46:31.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:46:31.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:31.377 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:46:31.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:46:31.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:31.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:46:32.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:46:33.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:46:33.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:33.396 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:46:33.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:46:33.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:33.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:46:34.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:46:35.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:46:35.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:35.672 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:46:35.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:46:35.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:35.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:46:36.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:46:37.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:46:37.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:37.334 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:46:37.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:46:37.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:37.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:46:38.167 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:46:39.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:46:39.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:39.169 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T12:46:39.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T12:46:39.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:39.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:46:39.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:46:40.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:46:40.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:46:40.670 INFO:tasks.workunit.client.0.smithi005.stdout:19 2024-05-15T12:46:40.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-15T12:46:40.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:46:40.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:46:41.126 INFO:tasks.workunit.client.0.smithi005.stdout:osd.4 up in weight 1 up_from 41 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/53239628,v1:127.0.0.1:6835/53239628] [v2:127.0.0.1:6836/53239628,v1:127.0.0.1:6837/53239628] exists,up 3f468dd3-1b89-47be-bdd3-b69fc55b54b8 2024-05-15T12:46:41.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:46:41.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:46:41.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:46:41.127 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-15T12:46:41.127 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T12:46:41.127 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T12:46:41.127 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:46:41.128 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:46:41.128 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:46:41.128 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.0.asok 2024-05-15T12:46:41.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-15T12:46:41.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-osd.0.asok log flush 2024-05-15T12:46:41.242 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:46:41.255 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-15T12:46:41.261 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:43:34.224+0000 7f7d14e577c0 0 load: jerasure load: lrc 2024-05-15T12:46:41.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:520: TEST_ec_backfill_unfound: CEPH_ARGS='--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:46:41.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:522: TEST_ec_backfill_unfound: local poolname=pool-jerasure 2024-05-15T12:46:41.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:523: TEST_ec_backfill_unfound: create_erasure_coded_pool pool-jerasure 3 2 2024-05-15T12:46:41.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-15T12:46:41.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-15T12:46:41.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-05-15T12:46:41.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-15T12:46:41.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-05-15T12:46:41.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-15T12:46:41.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-05-15T12:46:42.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-15T12:46:42.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-15T12:46:44.129 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-jerasure' already exists 2024-05-15T12:46:44.147 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T12:46:45.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-15T12:46:45.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T12:46:45.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T12:46:45.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T12:46:45.150 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T12:46:45.151 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T12:46:45.151 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T12:46:45.151 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T12:46:45.151 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T12:46:45.152 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T12:46:45.277 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T12:46:45.277 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T12:46:45.277 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T12:46:45.277 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T12:46:45.278 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T12:46:45.278 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T12:46:45.736 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T12:46:45.736 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:46:45.736 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T12:46:45.736 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:46:45.737 INFO:tasks.workunit.client.0.smithi005.stderr:4' 2024-05-15T12:46:45.737 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T12:46:45.737 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:46:45.737 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T12:46:46.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803818 2024-05-15T12:46:46.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803818 2024-05-15T12:46:46.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803818' 2024-05-15T12:46:46.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:46:46.013 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T12:46:46.191 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574881 2024-05-15T12:46:46.191 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574881 2024-05-15T12:46:46.191 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803818 1-55834574881' 2024-05-15T12:46:46.191 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:46:46.191 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T12:46:46.351 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345947 2024-05-15T12:46:46.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345947 2024-05-15T12:46:46.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803818 1-55834574881 2-85899345947' 2024-05-15T12:46:46.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:46:46.352 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T12:46:46.498 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018900 2024-05-15T12:46:46.498 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018900 2024-05-15T12:46:46.498 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803818 1-55834574881 2-85899345947 3-128849018900' 2024-05-15T12:46:46.499 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:46:46.499 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T12:46:46.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659143 2024-05-15T12:46:46.650 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659143 2024-05-15T12:46:46.650 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803818 1-55834574881 2-85899345947 3-128849018900 4-176093659143' 2024-05-15T12:46:46.650 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:46:46.650 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803818 2024-05-15T12:46:46.650 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:46:46.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T12:46:46.652 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803818 2024-05-15T12:46:46.652 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:46:46.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803818 2024-05-15T12:46:46.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803818' 2024-05-15T12:46:46.654 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 25769803818 2024-05-15T12:46:46.655 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T12:46:47.069 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803818 -lt 25769803818 2024-05-15T12:46:47.069 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:46:47.070 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574881 2024-05-15T12:46:47.070 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:46:47.071 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T12:46:47.072 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:46:47.072 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574881 2024-05-15T12:46:47.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574881 2024-05-15T12:46:47.074 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 55834574881 2024-05-15T12:46:47.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574881' 2024-05-15T12:46:47.074 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T12:46:47.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574882 -lt 55834574881 2024-05-15T12:46:47.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:46:47.523 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345947 2024-05-15T12:46:47.524 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:46:47.525 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T12:46:47.526 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345947 2024-05-15T12:46:47.526 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:46:47.528 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345947 2024-05-15T12:46:47.528 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 85899345947 2024-05-15T12:46:47.529 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345947' 2024-05-15T12:46:47.529 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T12:46:47.958 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345947 -lt 85899345947 2024-05-15T12:46:47.958 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:46:47.959 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-128849018900 2024-05-15T12:46:47.959 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:46:47.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T12:46:47.962 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-128849018900 2024-05-15T12:46:47.962 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:46:47.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018900 2024-05-15T12:46:47.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 128849018900' 2024-05-15T12:46:47.964 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 128849018900 2024-05-15T12:46:47.964 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T12:46:48.407 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018900 -lt 128849018900 2024-05-15T12:46:48.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:46:48.408 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-176093659143 2024-05-15T12:46:48.408 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:46:48.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T12:46:48.410 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-176093659143 2024-05-15T12:46:48.411 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:46:48.412 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659143 2024-05-15T12:46:48.412 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.4 seq 176093659143 2024-05-15T12:46:48.412 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 176093659143' 2024-05-15T12:46:48.412 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T12:46:48.907 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659143 -lt 176093659143 2024-05-15T12:46:48.908 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T12:46:48.908 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:46:48.908 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:46:49.466 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T12:46:49.467 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T12:46:49.468 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T12:46:49.468 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T12:46:49.468 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T12:46:49.468 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T12:46:49.468 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T12:46:49.468 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T12:46:49.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T12:46:49.940 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T12:46:49.940 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:46:49.941 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:46:50.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T12:46:50.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T12:46:50.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T12:46:50.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:525: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-05-15T12:46:50.887 INFO:tasks.workunit.client.0.smithi005.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-05-15T12:46:50.887 INFO:tasks.workunit.client.0.smithi005.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T12:46:45.560955+0000 0'0 50:15 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-05-15T12:46:43.900220+0000 0'0 2024-05-15T12:46:43.900220+0000 0 0 periodic scrub scheduled @ 2024-05-16T19:41:54.351336+0000 0 0 2024-05-15T12:46:50.887 INFO:tasks.workunit.client.0.smithi005.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T12:46:42.895695+0000 0'0 50:21 [4,1,2] 4 [4,1,2] 4 0'0 2024-05-15T12:43:24.211528+0000 0'0 2024-05-15T12:43:24.211528+0000 0 0 periodic scrub scheduled @ 2024-05-16T15:21:28.718205+0000 0 0 2024-05-15T12:46:50.887 INFO:tasks.workunit.client.0.smithi005.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T12:46:42.081784+0000 0'0 49:109 [0,4,1] 0 [0,4,1] 0 0'0 2024-05-15T12:43:24.211528+0000 0'0 2024-05-15T12:43:24.211528+0000 0 0 periodic scrub scheduled @ 2024-05-16T13:32:41.365959+0000 0 0 2024-05-15T12:46:50.887 INFO:tasks.workunit.client.0.smithi005.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T12:46:43.195626+0000 0'0 50:21 [4,3,0] 4 [4,3,0] 4 0'0 2024-05-15T12:43:24.211528+0000 0'0 2024-05-15T12:43:24.211528+0000 0 0 periodic scrub scheduled @ 2024-05-16T17:49:08.751042+0000 0 0 2024-05-15T12:46:50.887 INFO:tasks.workunit.client.0.smithi005.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T12:45:21.343336+0000 0'0 49:78 [1,0,2] 1 [1,0,2] 1 0'0 2024-05-15T12:43:24.211528+0000 0'0 2024-05-15T12:43:24.211528+0000 0 0 periodic scrub scheduled @ 2024-05-16T18:11:47.318869+0000 0 0 2024-05-15T12:46:50.887 INFO:tasks.workunit.client.0.smithi005.stdout: 2024-05-15T12:46:50.887 INFO:tasks.workunit.client.0.smithi005.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-05-15T12:46:50.888 INFO:tasks.workunit.client.0.smithi005.stderr:dumped pgs 2024-05-15T12:46:50.903 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:527: TEST_ec_backfill_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2024-05-15T12:46:50.903 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-15T12:46:50.904 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-15T12:46:50.904 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-05-15T12:46:50.904 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T12:46:50.904 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-15T12:46:50.904 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T12:46:50.904 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-15T12:46:50.904 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T12:46:50.904 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-15T12:46:50.904 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T12:46:50.904 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-15T12:46:50.904 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-05-15T12:46:51.090 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:528: TEST_ec_backfill_unfound: get_primary pool-jerasure myobject 2024-05-15T12:46:51.090 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=pool-jerasure 2024-05-15T12:46:51.090 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=myobject 2024-05-15T12:46:51.090 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map pool-jerasure myobject 2024-05-15T12:46:51.090 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-05-15T12:46:51.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:528: TEST_ec_backfill_unfound: local primary=3 2024-05-15T12:46:51.565 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: get_osds pool-jerasure myobject 2024-05-15T12:46:51.566 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T12:46:51.566 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-05-15T12:46:51.566 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:46:51.566 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-05-15T12:46:51.985 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T12:46:51.985 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:46:51.985 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T12:46:51.985 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T12:46:51.985 INFO:tasks.workunit.client.0.smithi005.stderr:2' 2024-05-15T12:46:51.985 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2 2024-05-15T12:46:51.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: initial_osds=('3' '1' '4' '0' '2') 2024-05-15T12:46:51.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: local -a initial_osds 2024-05-15T12:46:51.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:531: TEST_ec_backfill_unfound: local last_osd=2 2024-05-15T12:46:51.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:532: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-05-15T12:46:51.986 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:46:51.986 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:46:51.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:46:51.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:46:51.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:46:52.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:46:52.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:533: TEST_ec_backfill_unfound: ceph osd down 2 2024-05-15T12:46:53.165 INFO:tasks.workunit.client.0.smithi005.stderr:osd.2 is already down. 2024-05-15T12:46:53.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:534: TEST_ec_backfill_unfound: ceph osd out 2 2024-05-15T12:46:54.015 INFO:tasks.workunit.client.0.smithi005.stderr:osd.2 is already out. 2024-05-15T12:46:54.033 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:536: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-05-15T12:46:54.475 INFO:tasks.workunit.client.0.smithi005.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-05-15T12:46:54.475 INFO:tasks.workunit.client.0.smithi005.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T12:46:45.560955+0000 0'0 50:15 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-05-15T12:46:43.900220+0000 0'0 2024-05-15T12:46:43.900220+0000 0 0 periodic scrub scheduled @ 2024-05-16T19:41:54.351336+0000 0 0 2024-05-15T12:46:54.475 INFO:tasks.workunit.client.0.smithi005.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T12:46:42.895695+0000 0'0 50:21 [4,1,2] 4 [4,1,2] 4 0'0 2024-05-15T12:43:24.211528+0000 0'0 2024-05-15T12:43:24.211528+0000 0 0 periodic scrub scheduled @ 2024-05-16T15:21:28.718205+0000 0 0 2024-05-15T12:46:54.475 INFO:tasks.workunit.client.0.smithi005.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T12:46:42.081784+0000 0'0 53:117 [0,4,1] 0 [0,4,1] 0 0'0 2024-05-15T12:43:24.211528+0000 0'0 2024-05-15T12:43:24.211528+0000 0 0 periodic scrub scheduled @ 2024-05-16T13:32:41.365959+0000 0 0 2024-05-15T12:46:54.475 INFO:tasks.workunit.client.0.smithi005.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T12:46:43.195626+0000 0'0 50:21 [4,3,0] 4 [4,3,0] 4 0'0 2024-05-15T12:43:24.211528+0000 0'0 2024-05-15T12:43:24.211528+0000 0 0 periodic scrub scheduled @ 2024-05-16T17:49:08.751042+0000 0 0 2024-05-15T12:46:54.475 INFO:tasks.workunit.client.0.smithi005.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T12:45:21.343336+0000 0'0 50:80 [1,0,2] 1 [1,0,2] 1 0'0 2024-05-15T12:43:24.211528+0000 0'0 2024-05-15T12:43:24.211528+0000 0 0 periodic scrub scheduled @ 2024-05-16T18:11:47.318869+0000 0 0 2024-05-15T12:46:54.475 INFO:tasks.workunit.client.0.smithi005.stdout: 2024-05-15T12:46:54.475 INFO:tasks.workunit.client.0.smithi005.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-05-15T12:46:54.476 INFO:tasks.workunit.client.0.smithi005.stderr:dumped pgs 2024-05-15T12:46:54.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:538: TEST_ec_backfill_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2024-05-15T12:46:54.493 INFO:tasks.workunit.client.0.smithi005.stderr:4+0 records in 2024-05-15T12:46:54.493 INFO:tasks.workunit.client.0.smithi005.stderr:4+0 records out 2024-05-15T12:46:54.493 INFO:tasks.workunit.client.0.smithi005.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000192579 s, 21.3 MB/s 2024-05-15T12:46:54.494 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: seq 1 300 2024-05-15T12:46:54.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:46:54.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2024-05-15T12:46:54.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:46:54.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2024-05-15T12:46:55.470 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:46:55.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2024-05-15T12:46:55.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:46:55.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2024-05-15T12:46:56.041 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:46:56.041 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2024-05-15T12:46:56.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:46:56.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2024-05-15T12:46:56.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:46:56.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2024-05-15T12:46:56.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:46:56.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2024-05-15T12:46:57.131 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:46:57.131 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2024-05-15T12:46:57.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:46:57.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2024-05-15T12:46:57.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:46:57.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2024-05-15T12:46:58.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:46:58.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2024-05-15T12:46:58.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:46:58.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2024-05-15T12:46:58.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:46:58.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2024-05-15T12:46:58.833 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:46:58.833 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2024-05-15T12:46:59.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:46:59.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2024-05-15T12:46:59.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:46:59.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2024-05-15T12:46:59.847 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:46:59.847 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:00.140 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:00.140 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:00.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:00.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:00.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:00.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:01.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:01.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:01.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:01.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:01.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:01.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:01.817 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:01.817 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:02.094 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:02.094 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:02.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:02.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:02.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:02.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:02.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:02.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:03.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:03.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:03.478 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:03.478 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:03.781 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:03.781 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:04.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:04.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:04.434 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:04.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:04.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:04.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:04.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:04.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:05.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:05.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:05.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:05.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:05.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:05.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:06.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:06.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:06.412 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:06.413 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:06.732 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:06.732 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:07.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:07.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:07.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:07.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:07.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:07.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:07.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:07.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:08.140 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:08.140 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:08.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:08.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:08.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:08.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:09.021 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:09.021 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:09.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:09.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:09.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:09.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:09.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:09.835 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:10.127 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:10.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:10.437 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:10.437 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:10.722 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:10.722 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:11.021 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:11.022 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:11.326 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:11.326 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:11.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:11.620 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:11.914 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:11.914 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:12.206 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:12.206 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:12.499 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:12.499 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:12.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:12.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:13.036 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:13.036 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:13.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:13.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:13.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:13.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:13.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:13.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:14.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:14.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:14.511 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:14.511 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:14.780 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:14.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:15.065 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:15.065 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:15.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:15.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:15.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:15.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:15.878 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:15.878 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:16.197 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:16.197 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:16.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:16.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:16.725 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:16.725 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:17.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:17.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:17.345 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:17.345 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:17.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:17.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:17.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:17.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:18.167 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:18.167 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:18.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:18.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:18.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:18.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:18.997 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:18.997 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:19.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:19.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:19.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:19.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:19.868 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:19.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:20.120 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:20.120 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:20.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:20.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:20.715 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:20.716 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:20.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:20.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:21.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:21.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:21.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:21.556 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:21.839 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:21.840 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:22.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:22.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:22.392 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:22.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:22.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:22.687 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:22.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:22.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:23.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:23.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:23.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:23.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj101 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:23.802 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:23.802 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj102 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:24.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:24.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj103 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:24.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:24.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj104 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:24.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:24.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj105 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:24.992 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:24.992 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj106 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:25.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:25.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj107 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:25.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:25.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj108 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:25.788 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:25.788 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj109 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:26.091 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:26.091 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj110 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:26.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:26.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj111 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:26.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:26.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj112 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:26.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:26.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj113 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:27.222 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:27.222 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj114 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:27.508 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:27.508 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj115 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:27.808 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:27.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj116 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:28.068 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:28.069 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj117 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:28.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:28.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj118 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:28.596 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:28.596 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj119 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:28.865 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:28.865 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj120 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:29.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:29.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj121 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:29.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:29.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj122 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:30.021 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:30.022 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj123 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:30.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:30.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj124 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:30.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:30.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj125 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:30.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:30.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj126 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:31.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:31.088 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj127 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:31.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:31.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj128 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:31.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:31.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj129 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:31.949 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:31.949 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj130 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:32.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:32.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj131 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:32.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:32.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj132 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:32.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:32.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj133 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:33.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:33.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj134 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:33.385 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:33.386 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj135 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:33.677 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:33.677 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj136 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:33.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:33.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj137 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:34.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:34.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj138 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:34.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:34.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj139 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:34.716 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:34.716 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj140 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:35.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:35.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj141 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:35.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:35.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj142 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:35.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:35.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj143 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:36.024 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:36.024 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj144 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:36.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:36.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj145 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:36.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:36.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj146 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:36.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:36.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj147 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:37.123 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:37.123 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj148 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:37.458 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:37.459 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj149 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:37.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:37.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj150 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:37.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:37.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj151 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:38.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:38.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj152 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:38.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:38.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj153 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:38.808 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:38.808 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj154 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:39.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:39.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj155 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:39.386 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:39.386 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj156 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:39.646 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:39.646 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj157 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:39.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:39.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj158 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:40.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:40.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj159 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:40.510 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:40.510 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj160 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:40.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:40.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj161 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:41.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:41.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj162 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:41.406 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:41.407 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj163 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:41.725 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:41.725 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj164 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:41.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:41.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj165 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:42.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:42.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj166 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:42.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:42.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj167 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:42.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:42.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj168 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:43.066 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:43.067 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj169 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:43.351 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:43.351 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj170 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:43.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:43.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj171 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:43.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:43.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj172 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:44.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:44.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj173 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:44.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:44.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj174 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:44.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:44.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj175 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:44.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:44.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj176 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:45.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:45.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj177 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:45.516 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:45.516 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj178 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:45.802 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:45.802 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj179 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:46.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:46.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj180 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:46.372 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:46.373 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj181 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:46.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:46.650 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj182 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:46.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:46.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj183 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:47.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:47.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj184 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:47.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:47.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj185 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:47.789 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:47.789 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj186 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:48.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:48.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj187 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:48.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:48.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj188 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:48.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:48.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj189 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:48.915 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:48.915 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj190 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:49.226 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:49.226 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj191 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:49.493 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:49.493 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj192 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:49.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:49.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj193 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:50.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:50.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj194 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:50.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:50.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj195 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:50.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:50.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj196 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:50.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:50.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj197 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:51.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:51.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj198 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:51.553 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:51.553 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj199 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:51.838 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:51.838 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj200 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:52.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:52.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj201 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:52.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:52.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj202 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:52.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:52.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj203 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:52.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:52.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj204 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:53.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:53.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj205 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:53.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:53.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj206 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:53.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:53.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj207 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:54.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:54.103 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj208 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:54.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:54.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj209 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:54.647 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:54.647 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj210 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:54.891 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:54.891 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj211 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:55.216 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:55.216 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj212 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:55.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:55.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj213 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:55.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:55.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj214 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:56.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:56.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj215 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:56.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:56.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj216 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:56.717 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:56.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj217 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:56.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:56.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj218 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:57.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:57.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj219 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:57.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:57.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj220 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:57.867 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:57.867 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj221 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:58.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:58.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj222 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:58.419 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:58.419 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj223 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:58.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:58.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj224 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:59.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:59.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj225 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:59.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:59.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj226 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:59.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:59.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj227 td/test-erasure-eio/ORIGINAL 2024-05-15T12:47:59.870 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:47:59.870 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj228 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:00.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:00.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj229 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:00.482 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:00.482 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj230 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:00.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:00.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj231 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:01.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:01.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj232 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:01.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:01.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj233 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:01.840 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:01.840 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj234 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:02.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:02.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj235 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:02.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:02.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj236 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:02.678 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:02.678 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj237 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:02.948 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:02.948 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj238 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:03.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:03.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj239 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:03.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:03.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj240 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:03.751 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:03.751 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj241 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:04.044 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:04.044 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj242 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:04.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:04.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj243 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:04.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:04.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj244 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:04.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:04.900 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj245 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:05.211 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:05.211 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj246 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:05.512 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:05.512 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj247 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:05.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:05.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj248 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:06.187 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:06.187 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj249 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:06.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:06.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj250 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:06.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:06.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj251 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:07.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:07.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj252 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:07.371 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:07.372 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj253 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:07.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:07.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj254 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:07.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:07.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj255 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:08.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:08.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj256 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:08.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:08.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj257 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:08.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:08.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj258 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:09.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:09.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj259 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:09.385 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:09.385 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj260 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:09.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:09.653 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj261 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:09.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:09.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj262 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:10.230 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:10.231 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj263 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:10.498 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:10.499 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj264 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:10.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:10.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj265 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:11.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:11.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj266 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:11.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:11.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj267 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:11.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:11.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj268 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:11.941 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:11.941 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj269 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:12.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:12.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj270 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:12.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:12.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj271 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:12.780 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:12.780 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj272 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:13.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:13.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj273 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:13.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:13.400 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj274 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:13.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:13.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj275 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:13.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:13.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj276 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:14.187 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:14.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj277 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:14.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:14.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj278 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:14.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:14.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj279 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:15.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:15.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj280 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:15.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:15.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj281 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:15.562 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:15.563 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj282 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:15.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:15.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj283 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:16.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:16.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj284 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:16.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:16.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj285 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:16.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:16.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj286 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:16.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:16.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj287 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:17.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:17.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj288 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:17.609 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:17.609 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj289 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:17.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:17.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj290 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:18.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:18.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj291 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:18.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:18.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj292 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:18.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:18.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj293 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:18.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:18.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj294 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:19.218 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:19.218 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj295 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:19.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:19.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj296 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:19.789 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:19.789 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj297 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:20.041 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:20.041 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj298 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:20.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:20.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj299 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:20.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:48:20.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj300 td/test-erasure-eio/ORIGINAL 2024-05-15T12:48:20.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:544: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 0 2024-05-15T12:48:20.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-15T12:48:20.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-15T12:48:20.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-15T12:48:20.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-15T12:48:20.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-15T12:48:20.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-15T12:48:20.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj250 2024-05-15T12:48:20.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-15T12:48:20.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-15T12:48:20.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-15T12:48:20.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-05-15T12:48:20.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-15T12:48:20.814 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj250 2024-05-15T12:48:20.815 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T12:48:20.815 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-05-15T12:48:20.815 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj250 2024-05-15T12:48:20.815 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:48:21.268 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T12:48:21.268 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:48:21.268 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T12:48:21.269 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T12:48:21.269 INFO:tasks.workunit.client.0.smithi005.stderr:2147483647' 2024-05-15T12:48:21.269 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-05-15T12:48:21.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-05-15T12:48:21.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-15T12:48:21.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-05-15T12:48:21.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-15T12:48:21.269 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-05-15T12:48:21.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-15T12:48:21.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-05-15T12:48:21.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-15T12:48:21.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-05-15T12:48:21.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-15T12:48:21.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-15T12:48:21.271 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-15T12:48:21.271 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-05-15T12:48:21.271 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-15T12:48:21.271 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-15T12:48:21.271 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:48:21.272 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:48:21.272 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:48:21.272 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.3.asok 2024-05-15T12:48:21.272 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.80702/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-05-15T12:48:21.395 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-15T12:48:21.395 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-15T12:48:21.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-15T12:48:21.396 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-05-15T12:48:21.396 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-15T12:48:21.397 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-15T12:48:21.397 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:48:21.397 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:48:21.397 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:48:21.397 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.3.asok 2024-05-15T12:48:21.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-15T12:48:21.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-osd.3.asok injectdataerr pool-jerasure obj250 0 2024-05-15T12:48:21.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:545: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 1 2024-05-15T12:48:21.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-15T12:48:21.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-15T12:48:21.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-15T12:48:21.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-15T12:48:21.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-15T12:48:21.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-15T12:48:21.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj250 2024-05-15T12:48:21.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-15T12:48:21.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-15T12:48:21.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-15T12:48:21.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-05-15T12:48:21.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-15T12:48:21.522 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj250 2024-05-15T12:48:21.522 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T12:48:21.522 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-05-15T12:48:21.522 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj250 2024-05-15T12:48:21.523 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:48:21.933 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T12:48:21.933 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:48:21.933 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T12:48:21.933 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T12:48:21.933 INFO:tasks.workunit.client.0.smithi005.stderr:2147483647' 2024-05-15T12:48:21.934 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-05-15T12:48:21.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-05-15T12:48:21.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-15T12:48:21.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-05-15T12:48:21.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-15T12:48:21.934 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/1/type 2024-05-15T12:48:21.935 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-15T12:48:21.935 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-05-15T12:48:21.935 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-15T12:48:21.935 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-05-15T12:48:21.935 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-15T12:48:21.935 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-15T12:48:21.936 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-15T12:48:21.936 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-05-15T12:48:21.936 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-05-15T12:48:21.936 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-05-15T12:48:21.937 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:48:21.937 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:48:21.937 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:48:21.937 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.1.asok 2024-05-15T12:48:21.937 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.80702/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-05-15T12:48:22.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-15T12:48:22.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-15T12:48:22.064 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-15T12:48:22.064 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-05-15T12:48:22.065 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-05-15T12:48:22.065 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-05-15T12:48:22.065 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:48:22.065 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:48:22.065 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:48:22.065 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.1.asok 2024-05-15T12:48:22.066 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-15T12:48:22.066 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-osd.1.asok injectdataerr pool-jerasure obj250 1 2024-05-15T12:48:22.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:547: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2024-05-15T12:48:22.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-15T12:48:22.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-15T12:48:22.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-05-15T12:48:22.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-15T12:48:22.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-05-15T12:48:22.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:48:22.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:48:22.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:48:22.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:48:22.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-15T12:48:22.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-15T12:48:22.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-15T12:48:22.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-15T12:48:22.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:48:22.189 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-15T12:48:22.189 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:48:22.189 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:48:22.189 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:48:22.189 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:48:22.189 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:48:22.190 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:48:22.190 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:48:22.190 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:48:22.190 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:48:22.190 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:48:22.190 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:48:22.190 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:48:22.190 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:48:22.190 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:48:22.190 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-15T12:48:22.191 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-15T12:48:22.191 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-05-15T12:48:22.191 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.2 2024-05-15T12:48:22.191 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-05-15T12:48:22.191 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:48:22.191 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-05-15T12:48:22.192 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-05-15T12:48:22.192 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-15T12:48:22.192 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-15T12:48:22.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-15T12:48:22.223 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:48:22.234+0000 7f5457cb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:48:22.226 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:48:22.238+0000 7f5457cb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:48:22.227 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:48:22.238+0000 7f5457cb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:48:22.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-05-15T12:48:22.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:48:22.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-15T12:48:22.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:48:22.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:48:22.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:48:22.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:48:22.611 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:48:22.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:48:22.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:48:22.763 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:48:22.774+0000 7f5457cb67c0 -1 Falling back to public interface 2024-05-15T12:48:23.038 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:48:24.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:48:24.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:48:24.039 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:48:24.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:48:24.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:48:24.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:48:24.453 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:48:25.455 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:48:25.455 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:48:25.455 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:48:25.455 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:48:25.456 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:48:25.456 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:48:25.894 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:48:26.125 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:48:26.138+0000 7f5457cb67c0 -1 osd.2 50 log_to_monitors true 2024-05-15T12:48:26.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:48:26.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:48:26.896 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:48:26.896 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:48:26.896 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:48:26.896 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:48:27.361 INFO:tasks.workunit.client.0.smithi005.stdout:osd.2 up out weight 0 up_from 58 up_thru 21 down_at 51 last_clean_interval [20,50) [v2:127.0.0.1:6818/2603105505,v1:127.0.0.1:6819/2603105505] [v2:127.0.0.1:6820/2603105505,v1:127.0.0.1:6821/2603105505] exists,up 6a695c3c-f227-4356-a06e-53f491f74662 2024-05-15T12:48:27.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:48:27.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:48:27.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:48:27.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:548: TEST_ec_backfill_unfound: ceph osd in 2 2024-05-15T12:48:28.878 INFO:tasks.workunit.client.0.smithi005.stderr:osd.2 is already in. 2024-05-15T12:48:28.905 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: sleep 15 2024-05-15T12:48:43.908 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: seq 1 240 2024-05-15T12:48:43.910 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-15T12:48:43.910 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-15T12:48:43.910 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-15T12:48:43.911 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-15T12:48:43.911 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-15T12:48:43.911 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-15T12:48:44.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:44.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:44.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-15T12:48:44.331 INFO:tasks.workunit.client.0.smithi005.stdout:active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:44.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-15T12:48:44.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:44.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-15T12:48:45.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-15T12:48:45.333 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-15T12:48:45.333 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-15T12:48:45.334 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-15T12:48:45.334 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-15T12:48:45.334 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-15T12:48:45.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:45.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:45.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-15T12:48:45.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-15T12:48:45.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:45.765 INFO:tasks.workunit.client.0.smithi005.stdout:active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:45.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-15T12:48:46.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-15T12:48:46.767 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-15T12:48:46.767 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-15T12:48:46.767 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-15T12:48:46.768 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-15T12:48:46.768 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-15T12:48:47.247 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:47.247 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:47.247 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-15T12:48:47.249 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-15T12:48:47.250 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:47.250 INFO:tasks.workunit.client.0.smithi005.stdout:active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:47.250 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-15T12:48:48.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-15T12:48:48.252 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-15T12:48:48.252 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-15T12:48:48.252 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-15T12:48:48.253 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-15T12:48:48.253 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-15T12:48:48.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:48.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:48.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-15T12:48:48.676 INFO:tasks.workunit.client.0.smithi005.stdout:active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:48.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-15T12:48:48.677 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:48.677 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-15T12:48:49.678 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-15T12:48:49.678 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-15T12:48:49.678 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-15T12:48:49.679 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-15T12:48:49.679 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-15T12:48:49.679 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-15T12:48:50.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:50.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:50.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-15T12:48:50.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-15T12:48:50.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:50.110 INFO:tasks.workunit.client.0.smithi005.stdout:active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:50.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-15T12:48:51.111 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-15T12:48:51.112 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-15T12:48:51.112 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-15T12:48:51.112 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-15T12:48:51.112 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-15T12:48:51.113 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-15T12:48:51.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:51.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:51.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-15T12:48:51.539 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-15T12:48:51.539 INFO:tasks.workunit.client.0.smithi005.stdout:active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:51.539 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:51.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-15T12:48:52.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-15T12:48:52.542 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-15T12:48:52.542 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-15T12:48:52.542 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-15T12:48:52.542 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-15T12:48:52.542 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-15T12:48:52.965 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:52.965 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:52.965 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-15T12:48:52.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-15T12:48:52.967 INFO:tasks.workunit.client.0.smithi005.stdout:active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:52.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:52.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-15T12:48:53.968 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-15T12:48:53.969 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-15T12:48:53.969 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-15T12:48:53.969 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-15T12:48:53.969 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-15T12:48:53.970 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-15T12:48:54.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:54.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:54.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-15T12:48:54.395 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-15T12:48:54.395 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:54.395 INFO:tasks.workunit.client.0.smithi005.stdout:active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:54.395 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-15T12:48:55.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-15T12:48:55.397 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-15T12:48:55.397 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-15T12:48:55.397 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-15T12:48:55.398 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-15T12:48:55.398 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-15T12:48:55.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:55.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:55.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-15T12:48:55.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-15T12:48:55.828 INFO:tasks.workunit.client.0.smithi005.stdout:active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:55.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:55.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-15T12:48:56.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-15T12:48:56.830 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-15T12:48:56.830 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-15T12:48:56.831 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-15T12:48:56.831 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-15T12:48:56.831 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-15T12:48:57.255 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:57.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:57.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-15T12:48:57.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-15T12:48:57.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:57.258 INFO:tasks.workunit.client.0.smithi005.stdout:active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:57.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-15T12:48:58.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-15T12:48:58.260 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-15T12:48:58.260 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-15T12:48:58.260 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-15T12:48:58.260 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-15T12:48:58.261 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-15T12:48:58.677 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:58.677 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:58.677 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-15T12:48:58.679 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-15T12:48:58.679 INFO:tasks.workunit.client.0.smithi005.stdout:active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:58.679 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:48:58.679 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-15T12:48:59.680 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-15T12:48:59.681 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-15T12:48:59.681 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-15T12:48:59.681 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-15T12:48:59.682 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-15T12:48:59.682 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-15T12:49:00.103 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-15T12:49:00.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:49:00.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-15T12:49:00.105 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-15T12:49:00.105 INFO:tasks.workunit.client.0.smithi005.stdout:active+undersized+degraded+remapped+backfilling 2024-05-15T12:49:00.105 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-15T12:49:00.105 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-15T12:49:01.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-15T12:49:01.107 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-15T12:49:01.107 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-15T12:49:01.107 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-15T12:49:01.108 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-15T12:49:01.108 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-15T12:49:01.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2024-05-15T12:49:01.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2024-05-15T12:49:01.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-15T12:49:01.551 INFO:tasks.workunit.client.0.smithi005.stdout:active+backfill_unfound+undersized+degraded+remapped 2024-05-15T12:49:01.552 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 0 = 0 ']' 2024-05-15T12:49:01.552 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: break 2024-05-15T12:49:01.552 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:562: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-05-15T12:49:01.946 INFO:tasks.workunit.client.0.smithi005.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-05-15T12:49:01.946 INFO:tasks.workunit.client.0.smithi005.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2024-05-15T12:48:55.762820+0000 56'301 63:899 [3,1,4,0,2] 3 [3,1,4,0,NONE] 3 0'0 2024-05-15T12:46:43.900220+0000 0'0 2024-05-15T12:46:43.900220+0000 0 0 no scrub is scheduled 0 0 2024-05-15T12:49:01.946 INFO:tasks.workunit.client.0.smithi005.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T12:48:29.529864+0000 0'0 63:58 [4,1,2] 4 [4,1,2] 4 0'0 2024-05-15T12:43:24.211528+0000 0'0 2024-05-15T12:43:24.211528+0000 0 0 periodic scrub scheduled @ 2024-05-16T22:36:20.715297+0000 0 0 2024-05-15T12:49:01.946 INFO:tasks.workunit.client.0.smithi005.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T12:46:42.081784+0000 0'0 63:136 [0,4,1] 0 [0,4,1] 0 0'0 2024-05-15T12:43:24.211528+0000 0'0 2024-05-15T12:43:24.211528+0000 0 0 periodic scrub scheduled @ 2024-05-16T13:32:41.365959+0000 0 0 2024-05-15T12:49:01.946 INFO:tasks.workunit.client.0.smithi005.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T12:46:43.195626+0000 0'0 63:46 [4,3,0] 4 [4,3,0] 4 0'0 2024-05-15T12:43:24.211528+0000 0'0 2024-05-15T12:43:24.211528+0000 0 0 periodic scrub scheduled @ 2024-05-16T17:49:08.751042+0000 0 0 2024-05-15T12:49:01.946 INFO:tasks.workunit.client.0.smithi005.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T12:48:29.738829+0000 0'0 63:120 [1,0,2] 1 [1,0,2] 1 0'0 2024-05-15T12:43:24.211528+0000 0'0 2024-05-15T12:43:24.211528+0000 0 0 periodic scrub scheduled @ 2024-05-17T00:07:34.820390+0000 0 0 2024-05-15T12:49:01.946 INFO:tasks.workunit.client.0.smithi005.stdout: 2024-05-15T12:49:01.946 INFO:tasks.workunit.client.0.smithi005.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-05-15T12:49:01.947 INFO:tasks.workunit.client.0.smithi005.stderr:dumped pgs 2024-05-15T12:49:01.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:563: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-05-15T12:49:01.961 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:49:01.961 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:49:01.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:49:01.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:49:01.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:49:02.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:49:02.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:564: TEST_ec_backfill_unfound: sleep 5 2024-05-15T12:49:07.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:566: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-05-15T12:49:07.677 INFO:tasks.workunit.client.0.smithi005.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-05-15T12:49:07.678 INFO:tasks.workunit.client.0.smithi005.stdout:2.0 301 1 303 0 1 1232896 0 0 101 200 101 active+recovery_unfound+undersized+degraded 2024-05-15T12:49:03.634918+0000 56'301 66:912 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2024-05-15T12:46:43.900220+0000 0'0 2024-05-15T12:46:43.900220+0000 0 0 no scrub is scheduled 0 0 2024-05-15T12:49:07.678 INFO:tasks.workunit.client.0.smithi005.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-05-15T12:49:03.455941+0000 0'0 66:66 [4,1] 4 [4,1] 4 0'0 2024-05-15T12:43:24.211528+0000 0'0 2024-05-15T12:43:24.211528+0000 0 0 no scrub is scheduled 0 0 2024-05-15T12:49:07.678 INFO:tasks.workunit.client.0.smithi005.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T12:46:42.081784+0000 0'0 66:141 [0,4,1] 0 [0,4,1] 0 0'0 2024-05-15T12:43:24.211528+0000 0'0 2024-05-15T12:43:24.211528+0000 0 0 periodic scrub scheduled @ 2024-05-16T13:32:41.365959+0000 0 0 2024-05-15T12:49:07.678 INFO:tasks.workunit.client.0.smithi005.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T12:46:43.195626+0000 0'0 66:52 [4,3,0] 4 [4,3,0] 4 0'0 2024-05-15T12:43:24.211528+0000 0'0 2024-05-15T12:43:24.211528+0000 0 0 periodic scrub scheduled @ 2024-05-16T17:49:08.751042+0000 0 0 2024-05-15T12:49:07.678 INFO:tasks.workunit.client.0.smithi005.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-05-15T12:49:03.395422+0000 0'0 66:129 [1,0] 1 [1,0] 1 0'0 2024-05-15T12:43:24.211528+0000 0'0 2024-05-15T12:43:24.211528+0000 0 0 no scrub is scheduled 0 0 2024-05-15T12:49:07.678 INFO:tasks.workunit.client.0.smithi005.stdout: 2024-05-15T12:49:07.678 INFO:tasks.workunit.client.0.smithi005.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-05-15T12:49:07.678 INFO:tasks.workunit.client.0.smithi005.stderr:dumped pgs 2024-05-15T12:49:07.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:567: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-05-15T12:49:07.848 INFO:tasks.workunit.client.0.smithi005.stdout:{ 2024-05-15T12:49:07.848 INFO:tasks.workunit.client.0.smithi005.stdout: "num_missing": 1, 2024-05-15T12:49:07.848 INFO:tasks.workunit.client.0.smithi005.stdout: "num_unfound": 1, 2024-05-15T12:49:07.848 INFO:tasks.workunit.client.0.smithi005.stdout: "objects": [ 2024-05-15T12:49:07.848 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T12:49:07.848 INFO:tasks.workunit.client.0.smithi005.stdout: "oid": { 2024-05-15T12:49:07.848 INFO:tasks.workunit.client.0.smithi005.stdout: "oid": "obj250", 2024-05-15T12:49:07.848 INFO:tasks.workunit.client.0.smithi005.stdout: "key": "", 2024-05-15T12:49:07.849 INFO:tasks.workunit.client.0.smithi005.stdout: "snapid": -2, 2024-05-15T12:49:07.849 INFO:tasks.workunit.client.0.smithi005.stdout: "hash": 2249616407, 2024-05-15T12:49:07.849 INFO:tasks.workunit.client.0.smithi005.stdout: "max": 0, 2024-05-15T12:49:07.849 INFO:tasks.workunit.client.0.smithi005.stdout: "pool": 2, 2024-05-15T12:49:07.849 INFO:tasks.workunit.client.0.smithi005.stdout: "namespace": "" 2024-05-15T12:49:07.849 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:07.849 INFO:tasks.workunit.client.0.smithi005.stdout: "need": "56'251", 2024-05-15T12:49:07.849 INFO:tasks.workunit.client.0.smithi005.stdout: "have": "0'0", 2024-05-15T12:49:07.849 INFO:tasks.workunit.client.0.smithi005.stdout: "flags": "none", 2024-05-15T12:49:07.849 INFO:tasks.workunit.client.0.smithi005.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-05-15T12:49:07.849 INFO:tasks.workunit.client.0.smithi005.stdout: "locations": [ 2024-05-15T12:49:07.849 INFO:tasks.workunit.client.0.smithi005.stdout: "0(3)", 2024-05-15T12:49:07.849 INFO:tasks.workunit.client.0.smithi005.stdout: "4(2)" 2024-05-15T12:49:07.849 INFO:tasks.workunit.client.0.smithi005.stdout: ] 2024-05-15T12:49:07.849 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T12:49:07.850 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:07.850 INFO:tasks.workunit.client.0.smithi005.stdout: "state": "NotRecovering", 2024-05-15T12:49:07.850 INFO:tasks.workunit.client.0.smithi005.stdout: "available_might_have_unfound": true, 2024-05-15T12:49:07.850 INFO:tasks.workunit.client.0.smithi005.stdout: "might_have_unfound": [ 2024-05-15T12:49:07.850 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T12:49:07.850 INFO:tasks.workunit.client.0.smithi005.stdout: "osd": "2(4)", 2024-05-15T12:49:07.850 INFO:tasks.workunit.client.0.smithi005.stdout: "status": "osd is down" 2024-05-15T12:49:07.850 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T12:49:07.850 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:07.850 INFO:tasks.workunit.client.0.smithi005.stdout: "more": false 2024-05-15T12:49:07.850 INFO:tasks.workunit.client.0.smithi005.stdout:} 2024-05-15T12:49:07.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:568: TEST_ec_backfill_unfound: ceph pg 2.0 query 2024-05-15T12:49:08.013 INFO:tasks.workunit.client.0.smithi005.stdout:{ 2024-05-15T12:49:08.014 INFO:tasks.workunit.client.0.smithi005.stdout: "snap_trimq": "[]", 2024-05-15T12:49:08.014 INFO:tasks.workunit.client.0.smithi005.stdout: "snap_trimq_len": 0, 2024-05-15T12:49:08.014 INFO:tasks.workunit.client.0.smithi005.stdout: "state": "active+recovery_unfound+undersized+degraded", 2024-05-15T12:49:08.014 INFO:tasks.workunit.client.0.smithi005.stdout: "epoch": 66, 2024-05-15T12:49:08.014 INFO:tasks.workunit.client.0.smithi005.stdout: "up": [ 2024-05-15T12:49:08.014 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T12:49:08.014 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T12:49:08.014 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T12:49:08.014 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T12:49:08.014 INFO:tasks.workunit.client.0.smithi005.stdout: 2147483647 2024-05-15T12:49:08.014 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.014 INFO:tasks.workunit.client.0.smithi005.stdout: "acting": [ 2024-05-15T12:49:08.014 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T12:49:08.014 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T12:49:08.015 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T12:49:08.015 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T12:49:08.015 INFO:tasks.workunit.client.0.smithi005.stdout: 2147483647 2024-05-15T12:49:08.015 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.015 INFO:tasks.workunit.client.0.smithi005.stdout: "acting_recovery_backfill": [ 2024-05-15T12:49:08.015 INFO:tasks.workunit.client.0.smithi005.stdout: "0(3)", 2024-05-15T12:49:08.015 INFO:tasks.workunit.client.0.smithi005.stdout: "1(1)", 2024-05-15T12:49:08.015 INFO:tasks.workunit.client.0.smithi005.stdout: "3(0)", 2024-05-15T12:49:08.015 INFO:tasks.workunit.client.0.smithi005.stdout: "4(2)" 2024-05-15T12:49:08.015 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.015 INFO:tasks.workunit.client.0.smithi005.stdout: "info": { 2024-05-15T12:49:08.015 INFO:tasks.workunit.client.0.smithi005.stdout: "pgid": "2.0s0", 2024-05-15T12:49:08.015 INFO:tasks.workunit.client.0.smithi005.stdout: "shared": "0", 2024-05-15T12:49:08.015 INFO:tasks.workunit.client.0.smithi005.stdout: "last_update": "56'301", 2024-05-15T12:49:08.015 INFO:tasks.workunit.client.0.smithi005.stdout: "last_complete": "56'250", 2024-05-15T12:49:08.015 INFO:tasks.workunit.client.0.smithi005.stdout: "log_tail": "56'200", 2024-05-15T12:49:08.016 INFO:tasks.workunit.client.0.smithi005.stdout: "last_user_version": 301, 2024-05-15T12:49:08.016 INFO:tasks.workunit.client.0.smithi005.stdout: "last_backfill": "MAX", 2024-05-15T12:49:08.016 INFO:tasks.workunit.client.0.smithi005.stdout: "purged_snaps": [], 2024-05-15T12:49:08.016 INFO:tasks.workunit.client.0.smithi005.stdout: "history": { 2024-05-15T12:49:08.016 INFO:tasks.workunit.client.0.smithi005.stdout: "epoch_created": 48, 2024-05-15T12:49:08.016 INFO:tasks.workunit.client.0.smithi005.stdout: "epoch_pool_created": 48, 2024-05-15T12:49:08.016 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_started": 66, 2024-05-15T12:49:08.016 INFO:tasks.workunit.client.0.smithi005.stdout: "last_interval_started": 64, 2024-05-15T12:49:08.016 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_clean": 49, 2024-05-15T12:49:08.016 INFO:tasks.workunit.client.0.smithi005.stdout: "last_interval_clean": 48, 2024-05-15T12:49:08.016 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_split": 0, 2024-05-15T12:49:08.016 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_marked_full": 0, 2024-05-15T12:49:08.016 INFO:tasks.workunit.client.0.smithi005.stdout: "same_up_since": 64, 2024-05-15T12:49:08.016 INFO:tasks.workunit.client.0.smithi005.stdout: "same_interval_since": 64, 2024-05-15T12:49:08.016 INFO:tasks.workunit.client.0.smithi005.stdout: "same_primary_since": 48, 2024-05-15T12:49:08.016 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub": "0'0", 2024-05-15T12:49:08.017 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.017 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub": "0'0", 2024-05-15T12:49:08.017 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.017 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.017 INFO:tasks.workunit.client.0.smithi005.stdout: "prior_readable_until_ub": 0 2024-05-15T12:49:08.017 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.017 INFO:tasks.workunit.client.0.smithi005.stdout: "stats": { 2024-05-15T12:49:08.017 INFO:tasks.workunit.client.0.smithi005.stdout: "version": "56'301", 2024-05-15T12:49:08.017 INFO:tasks.workunit.client.0.smithi005.stdout: "reported_seq": 913, 2024-05-15T12:49:08.017 INFO:tasks.workunit.client.0.smithi005.stdout: "reported_epoch": 66, 2024-05-15T12:49:08.017 INFO:tasks.workunit.client.0.smithi005.stdout: "state": "active+recovery_unfound+undersized+degraded", 2024-05-15T12:49:08.017 INFO:tasks.workunit.client.0.smithi005.stdout: "last_fresh": "2024-05-15T12:49:03.634918+0000", 2024-05-15T12:49:08.017 INFO:tasks.workunit.client.0.smithi005.stdout: "last_change": "2024-05-15T12:49:03.634918+0000", 2024-05-15T12:49:08.017 INFO:tasks.workunit.client.0.smithi005.stdout: "last_active": "2024-05-15T12:49:03.634918+0000", 2024-05-15T12:49:08.017 INFO:tasks.workunit.client.0.smithi005.stdout: "last_peered": "2024-05-15T12:49:03.634918+0000", 2024-05-15T12:49:08.017 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean": "2024-05-15T12:46:51.099810+0000", 2024-05-15T12:49:08.018 INFO:tasks.workunit.client.0.smithi005.stdout: "last_became_active": "2024-05-15T12:49:03.582752+0000", 2024-05-15T12:49:08.018 INFO:tasks.workunit.client.0.smithi005.stdout: "last_became_peered": "2024-05-15T12:49:03.582752+0000", 2024-05-15T12:49:08.018 INFO:tasks.workunit.client.0.smithi005.stdout: "last_unstale": "2024-05-15T12:49:03.634918+0000", 2024-05-15T12:49:08.018 INFO:tasks.workunit.client.0.smithi005.stdout: "last_undegraded": "2024-05-15T12:49:03.245368+0000", 2024-05-15T12:49:08.018 INFO:tasks.workunit.client.0.smithi005.stdout: "last_fullsized": "2024-05-15T12:49:03.244884+0000", 2024-05-15T12:49:08.018 INFO:tasks.workunit.client.0.smithi005.stdout: "mapping_epoch": 64, 2024-05-15T12:49:08.018 INFO:tasks.workunit.client.0.smithi005.stdout: "log_start": "56'200", 2024-05-15T12:49:08.018 INFO:tasks.workunit.client.0.smithi005.stdout: "ondisk_log_start": "56'200", 2024-05-15T12:49:08.018 INFO:tasks.workunit.client.0.smithi005.stdout: "created": 48, 2024-05-15T12:49:08.018 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_clean": 49, 2024-05-15T12:49:08.018 INFO:tasks.workunit.client.0.smithi005.stdout: "parent": "0.0", 2024-05-15T12:49:08.018 INFO:tasks.workunit.client.0.smithi005.stdout: "parent_split_bits": 0, 2024-05-15T12:49:08.018 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub": "0'0", 2024-05-15T12:49:08.018 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.018 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub": "0'0", 2024-05-15T12:49:08.019 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.019 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.019 INFO:tasks.workunit.client.0.smithi005.stdout: "objects_scrubbed": 0, 2024-05-15T12:49:08.019 INFO:tasks.workunit.client.0.smithi005.stdout: "log_size": 101, 2024-05-15T12:49:08.019 INFO:tasks.workunit.client.0.smithi005.stdout: "log_dups_size": 200, 2024-05-15T12:49:08.019 INFO:tasks.workunit.client.0.smithi005.stdout: "ondisk_log_size": 101, 2024-05-15T12:49:08.019 INFO:tasks.workunit.client.0.smithi005.stdout: "stats_invalid": false, 2024-05-15T12:49:08.019 INFO:tasks.workunit.client.0.smithi005.stdout: "dirty_stats_invalid": false, 2024-05-15T12:49:08.019 INFO:tasks.workunit.client.0.smithi005.stdout: "omap_stats_invalid": false, 2024-05-15T12:49:08.019 INFO:tasks.workunit.client.0.smithi005.stdout: "hitset_stats_invalid": false, 2024-05-15T12:49:08.019 INFO:tasks.workunit.client.0.smithi005.stdout: "hitset_bytes_stats_invalid": false, 2024-05-15T12:49:08.019 INFO:tasks.workunit.client.0.smithi005.stdout: "pin_stats_invalid": false, 2024-05-15T12:49:08.019 INFO:tasks.workunit.client.0.smithi005.stdout: "manifest_stats_invalid": false, 2024-05-15T12:49:08.019 INFO:tasks.workunit.client.0.smithi005.stdout: "snaptrimq_len": 0, 2024-05-15T12:49:08.020 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_duration": 0, 2024-05-15T12:49:08.020 INFO:tasks.workunit.client.0.smithi005.stdout: "scrub_schedule": "no scrub is scheduled", 2024-05-15T12:49:08.020 INFO:tasks.workunit.client.0.smithi005.stdout: "scrub_duration": 0, 2024-05-15T12:49:08.020 INFO:tasks.workunit.client.0.smithi005.stdout: "objects_trimmed": 0, 2024-05-15T12:49:08.020 INFO:tasks.workunit.client.0.smithi005.stdout: "snaptrim_duration": 0, 2024-05-15T12:49:08.020 INFO:tasks.workunit.client.0.smithi005.stdout: "stat_sum": { 2024-05-15T12:49:08.020 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes": 1232896, 2024-05-15T12:49:08.020 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects": 301, 2024-05-15T12:49:08.020 INFO:tasks.workunit.client.0.smithi005.stdout: "num_object_clones": 0, 2024-05-15T12:49:08.020 INFO:tasks.workunit.client.0.smithi005.stdout: "num_object_copies": 1505, 2024-05-15T12:49:08.020 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_missing_on_primary": 1, 2024-05-15T12:49:08.020 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_missing": 1, 2024-05-15T12:49:08.020 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_degraded": 303, 2024-05-15T12:49:08.021 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_misplaced": 0, 2024-05-15T12:49:08.021 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_unfound": 1, 2024-05-15T12:49:08.021 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_dirty": 301, 2024-05-15T12:49:08.021 INFO:tasks.workunit.client.0.smithi005.stdout: "num_whiteouts": 0, 2024-05-15T12:49:08.021 INFO:tasks.workunit.client.0.smithi005.stdout: "num_read": 0, 2024-05-15T12:49:08.021 INFO:tasks.workunit.client.0.smithi005.stdout: "num_read_kb": 0, 2024-05-15T12:49:08.021 INFO:tasks.workunit.client.0.smithi005.stdout: "num_write": 301, 2024-05-15T12:49:08.021 INFO:tasks.workunit.client.0.smithi005.stdout: "num_write_kb": 1204, 2024-05-15T12:49:08.021 INFO:tasks.workunit.client.0.smithi005.stdout: "num_scrub_errors": 0, 2024-05-15T12:49:08.021 INFO:tasks.workunit.client.0.smithi005.stdout: "num_shallow_scrub_errors": 0, 2024-05-15T12:49:08.021 INFO:tasks.workunit.client.0.smithi005.stdout: "num_deep_scrub_errors": 0, 2024-05-15T12:49:08.021 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_recovered": 0, 2024-05-15T12:49:08.021 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes_recovered": 0, 2024-05-15T12:49:08.021 INFO:tasks.workunit.client.0.smithi005.stdout: "num_keys_recovered": 0, 2024-05-15T12:49:08.022 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_omap": 0, 2024-05-15T12:49:08.022 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_hit_set_archive": 0, 2024-05-15T12:49:08.022 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes_hit_set_archive": 0, 2024-05-15T12:49:08.022 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush": 0, 2024-05-15T12:49:08.022 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_kb": 0, 2024-05-15T12:49:08.022 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict": 0, 2024-05-15T12:49:08.022 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_kb": 0, 2024-05-15T12:49:08.022 INFO:tasks.workunit.client.0.smithi005.stdout: "num_promote": 0, 2024-05-15T12:49:08.022 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_mode_high": 0, 2024-05-15T12:49:08.022 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_mode_low": 0, 2024-05-15T12:49:08.022 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_mode_some": 0, 2024-05-15T12:49:08.022 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_mode_full": 0, 2024-05-15T12:49:08.022 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_pinned": 0, 2024-05-15T12:49:08.022 INFO:tasks.workunit.client.0.smithi005.stdout: "num_legacy_snapsets": 0, 2024-05-15T12:49:08.022 INFO:tasks.workunit.client.0.smithi005.stdout: "num_large_omap_objects": 0, 2024-05-15T12:49:08.023 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_manifest": 0, 2024-05-15T12:49:08.023 INFO:tasks.workunit.client.0.smithi005.stdout: "num_omap_bytes": 0, 2024-05-15T12:49:08.023 INFO:tasks.workunit.client.0.smithi005.stdout: "num_omap_keys": 0, 2024-05-15T12:49:08.023 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_repaired": 0 2024-05-15T12:49:08.023 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.023 INFO:tasks.workunit.client.0.smithi005.stdout: "up": [ 2024-05-15T12:49:08.023 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T12:49:08.023 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T12:49:08.023 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T12:49:08.023 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T12:49:08.023 INFO:tasks.workunit.client.0.smithi005.stdout: 2147483647 2024-05-15T12:49:08.023 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.023 INFO:tasks.workunit.client.0.smithi005.stdout: "acting": [ 2024-05-15T12:49:08.023 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T12:49:08.023 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T12:49:08.023 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T12:49:08.023 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T12:49:08.024 INFO:tasks.workunit.client.0.smithi005.stdout: 2147483647 2024-05-15T12:49:08.024 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.024 INFO:tasks.workunit.client.0.smithi005.stdout: "avail_no_missing": [ 2024-05-15T12:49:08.024 INFO:tasks.workunit.client.0.smithi005.stdout: "0(3)", 2024-05-15T12:49:08.024 INFO:tasks.workunit.client.0.smithi005.stdout: "4(2)" 2024-05-15T12:49:08.024 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.024 INFO:tasks.workunit.client.0.smithi005.stdout: "object_location_counts": [ 2024-05-15T12:49:08.024 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T12:49:08.024 INFO:tasks.workunit.client.0.smithi005.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-05-15T12:49:08.024 INFO:tasks.workunit.client.0.smithi005.stdout: "objects": 300 2024-05-15T12:49:08.024 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.024 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T12:49:08.024 INFO:tasks.workunit.client.0.smithi005.stdout: "shards": "0(3),4(2)", 2024-05-15T12:49:08.024 INFO:tasks.workunit.client.0.smithi005.stdout: "objects": 1 2024-05-15T12:49:08.024 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T12:49:08.024 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.024 INFO:tasks.workunit.client.0.smithi005.stdout: "blocked_by": [], 2024-05-15T12:49:08.025 INFO:tasks.workunit.client.0.smithi005.stdout: "up_primary": 3, 2024-05-15T12:49:08.025 INFO:tasks.workunit.client.0.smithi005.stdout: "acting_primary": 3, 2024-05-15T12:49:08.025 INFO:tasks.workunit.client.0.smithi005.stdout: "purged_snaps": [] 2024-05-15T12:49:08.025 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.025 INFO:tasks.workunit.client.0.smithi005.stdout: "empty": 0, 2024-05-15T12:49:08.025 INFO:tasks.workunit.client.0.smithi005.stdout: "dne": 0, 2024-05-15T12:49:08.025 INFO:tasks.workunit.client.0.smithi005.stdout: "incomplete": 0, 2024-05-15T12:49:08.025 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_started": 66, 2024-05-15T12:49:08.025 INFO:tasks.workunit.client.0.smithi005.stdout: "hit_set_history": { 2024-05-15T12:49:08.025 INFO:tasks.workunit.client.0.smithi005.stdout: "current_last_update": "0'0", 2024-05-15T12:49:08.025 INFO:tasks.workunit.client.0.smithi005.stdout: "history": [] 2024-05-15T12:49:08.025 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T12:49:08.025 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.025 INFO:tasks.workunit.client.0.smithi005.stdout: "peer_info": [ 2024-05-15T12:49:08.025 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T12:49:08.025 INFO:tasks.workunit.client.0.smithi005.stdout: "peer": "0(3)", 2024-05-15T12:49:08.026 INFO:tasks.workunit.client.0.smithi005.stdout: "pgid": "2.0s3", 2024-05-15T12:49:08.026 INFO:tasks.workunit.client.0.smithi005.stdout: "shared": "3", 2024-05-15T12:49:08.026 INFO:tasks.workunit.client.0.smithi005.stdout: "last_update": "56'301", 2024-05-15T12:49:08.026 INFO:tasks.workunit.client.0.smithi005.stdout: "last_complete": "56'301", 2024-05-15T12:49:08.026 INFO:tasks.workunit.client.0.smithi005.stdout: "log_tail": "56'200", 2024-05-15T12:49:08.026 INFO:tasks.workunit.client.0.smithi005.stdout: "last_user_version": 301, 2024-05-15T12:49:08.026 INFO:tasks.workunit.client.0.smithi005.stdout: "last_backfill": "MAX", 2024-05-15T12:49:08.026 INFO:tasks.workunit.client.0.smithi005.stdout: "purged_snaps": [], 2024-05-15T12:49:08.026 INFO:tasks.workunit.client.0.smithi005.stdout: "history": { 2024-05-15T12:49:08.026 INFO:tasks.workunit.client.0.smithi005.stdout: "epoch_created": 48, 2024-05-15T12:49:08.026 INFO:tasks.workunit.client.0.smithi005.stdout: "epoch_pool_created": 48, 2024-05-15T12:49:08.026 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_started": 66, 2024-05-15T12:49:08.026 INFO:tasks.workunit.client.0.smithi005.stdout: "last_interval_started": 64, 2024-05-15T12:49:08.026 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_clean": 49, 2024-05-15T12:49:08.026 INFO:tasks.workunit.client.0.smithi005.stdout: "last_interval_clean": 48, 2024-05-15T12:49:08.027 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_split": 0, 2024-05-15T12:49:08.027 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_marked_full": 0, 2024-05-15T12:49:08.027 INFO:tasks.workunit.client.0.smithi005.stdout: "same_up_since": 64, 2024-05-15T12:49:08.027 INFO:tasks.workunit.client.0.smithi005.stdout: "same_interval_since": 64, 2024-05-15T12:49:08.027 INFO:tasks.workunit.client.0.smithi005.stdout: "same_primary_since": 48, 2024-05-15T12:49:08.027 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub": "0'0", 2024-05-15T12:49:08.027 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.027 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub": "0'0", 2024-05-15T12:49:08.027 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.027 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.027 INFO:tasks.workunit.client.0.smithi005.stdout: "prior_readable_until_ub": 0 2024-05-15T12:49:08.027 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.027 INFO:tasks.workunit.client.0.smithi005.stdout: "stats": { 2024-05-15T12:49:08.027 INFO:tasks.workunit.client.0.smithi005.stdout: "version": "56'301", 2024-05-15T12:49:08.027 INFO:tasks.workunit.client.0.smithi005.stdout: "reported_seq": 902, 2024-05-15T12:49:08.027 INFO:tasks.workunit.client.0.smithi005.stdout: "reported_epoch": 64, 2024-05-15T12:49:08.028 INFO:tasks.workunit.client.0.smithi005.stdout: "state": "active+undersized+degraded", 2024-05-15T12:49:08.028 INFO:tasks.workunit.client.0.smithi005.stdout: "last_fresh": "2024-05-15T12:48:20.822012+0000", 2024-05-15T12:49:08.028 INFO:tasks.workunit.client.0.smithi005.stdout: "last_change": "2024-05-15T12:46:53.562212+0000", 2024-05-15T12:49:08.028 INFO:tasks.workunit.client.0.smithi005.stdout: "last_active": "2024-05-15T12:48:20.822012+0000", 2024-05-15T12:49:08.028 INFO:tasks.workunit.client.0.smithi005.stdout: "last_peered": "2024-05-15T12:48:20.822012+0000", 2024-05-15T12:49:08.028 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean": "2024-05-15T12:46:51.099810+0000", 2024-05-15T12:49:08.028 INFO:tasks.workunit.client.0.smithi005.stdout: "last_became_active": "2024-05-15T12:46:53.562212+0000", 2024-05-15T12:49:08.028 INFO:tasks.workunit.client.0.smithi005.stdout: "last_became_peered": "2024-05-15T12:46:53.562212+0000", 2024-05-15T12:49:08.028 INFO:tasks.workunit.client.0.smithi005.stdout: "last_unstale": "2024-05-15T12:48:20.822012+0000", 2024-05-15T12:49:08.028 INFO:tasks.workunit.client.0.smithi005.stdout: "last_undegraded": "2024-05-15T12:46:53.235098+0000", 2024-05-15T12:49:08.028 INFO:tasks.workunit.client.0.smithi005.stdout: "last_fullsized": "2024-05-15T12:46:53.234482+0000", 2024-05-15T12:49:08.028 INFO:tasks.workunit.client.0.smithi005.stdout: "mapping_epoch": 64, 2024-05-15T12:49:08.028 INFO:tasks.workunit.client.0.smithi005.stdout: "log_start": "56'200", 2024-05-15T12:49:08.028 INFO:tasks.workunit.client.0.smithi005.stdout: "ondisk_log_start": "56'200", 2024-05-15T12:49:08.028 INFO:tasks.workunit.client.0.smithi005.stdout: "created": 48, 2024-05-15T12:49:08.028 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_clean": 49, 2024-05-15T12:49:08.029 INFO:tasks.workunit.client.0.smithi005.stdout: "parent": "0.0", 2024-05-15T12:49:08.029 INFO:tasks.workunit.client.0.smithi005.stdout: "parent_split_bits": 0, 2024-05-15T12:49:08.029 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub": "0'0", 2024-05-15T12:49:08.029 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.029 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub": "0'0", 2024-05-15T12:49:08.029 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.029 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.029 INFO:tasks.workunit.client.0.smithi005.stdout: "objects_scrubbed": 0, 2024-05-15T12:49:08.029 INFO:tasks.workunit.client.0.smithi005.stdout: "log_size": 101, 2024-05-15T12:49:08.029 INFO:tasks.workunit.client.0.smithi005.stdout: "log_dups_size": 200, 2024-05-15T12:49:08.029 INFO:tasks.workunit.client.0.smithi005.stdout: "ondisk_log_size": 101, 2024-05-15T12:49:08.030 INFO:tasks.workunit.client.0.smithi005.stdout: "stats_invalid": false, 2024-05-15T12:49:08.030 INFO:tasks.workunit.client.0.smithi005.stdout: "dirty_stats_invalid": false, 2024-05-15T12:49:08.030 INFO:tasks.workunit.client.0.smithi005.stdout: "omap_stats_invalid": false, 2024-05-15T12:49:08.030 INFO:tasks.workunit.client.0.smithi005.stdout: "hitset_stats_invalid": false, 2024-05-15T12:49:08.030 INFO:tasks.workunit.client.0.smithi005.stdout: "hitset_bytes_stats_invalid": false, 2024-05-15T12:49:08.030 INFO:tasks.workunit.client.0.smithi005.stdout: "pin_stats_invalid": false, 2024-05-15T12:49:08.030 INFO:tasks.workunit.client.0.smithi005.stdout: "manifest_stats_invalid": false, 2024-05-15T12:49:08.030 INFO:tasks.workunit.client.0.smithi005.stdout: "snaptrimq_len": 0, 2024-05-15T12:49:08.030 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_duration": 0, 2024-05-15T12:49:08.030 INFO:tasks.workunit.client.0.smithi005.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-05-16T18:05:59.846940+0000", 2024-05-15T12:49:08.030 INFO:tasks.workunit.client.0.smithi005.stdout: "scrub_duration": 0, 2024-05-15T12:49:08.030 INFO:tasks.workunit.client.0.smithi005.stdout: "objects_trimmed": 0, 2024-05-15T12:49:08.030 INFO:tasks.workunit.client.0.smithi005.stdout: "snaptrim_duration": 0, 2024-05-15T12:49:08.030 INFO:tasks.workunit.client.0.smithi005.stdout: "stat_sum": { 2024-05-15T12:49:08.030 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes": 1232896, 2024-05-15T12:49:08.031 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects": 301, 2024-05-15T12:49:08.031 INFO:tasks.workunit.client.0.smithi005.stdout: "num_object_clones": 0, 2024-05-15T12:49:08.031 INFO:tasks.workunit.client.0.smithi005.stdout: "num_object_copies": 1505, 2024-05-15T12:49:08.031 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_missing_on_primary": 0, 2024-05-15T12:49:08.031 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_missing": 0, 2024-05-15T12:49:08.031 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_degraded": 301, 2024-05-15T12:49:08.031 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_misplaced": 0, 2024-05-15T12:49:08.031 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_unfound": 0, 2024-05-15T12:49:08.031 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_dirty": 301, 2024-05-15T12:49:08.032 INFO:tasks.workunit.client.0.smithi005.stdout: "num_whiteouts": 0, 2024-05-15T12:49:08.032 INFO:tasks.workunit.client.0.smithi005.stdout: "num_read": 0, 2024-05-15T12:49:08.032 INFO:tasks.workunit.client.0.smithi005.stdout: "num_read_kb": 0, 2024-05-15T12:49:08.032 INFO:tasks.workunit.client.0.smithi005.stdout: "num_write": 301, 2024-05-15T12:49:08.032 INFO:tasks.workunit.client.0.smithi005.stdout: "num_write_kb": 1204, 2024-05-15T12:49:08.032 INFO:tasks.workunit.client.0.smithi005.stdout: "num_scrub_errors": 0, 2024-05-15T12:49:08.032 INFO:tasks.workunit.client.0.smithi005.stdout: "num_shallow_scrub_errors": 0, 2024-05-15T12:49:08.032 INFO:tasks.workunit.client.0.smithi005.stdout: "num_deep_scrub_errors": 0, 2024-05-15T12:49:08.032 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_recovered": 0, 2024-05-15T12:49:08.032 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes_recovered": 0, 2024-05-15T12:49:08.032 INFO:tasks.workunit.client.0.smithi005.stdout: "num_keys_recovered": 0, 2024-05-15T12:49:08.032 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_omap": 0, 2024-05-15T12:49:08.032 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_hit_set_archive": 0, 2024-05-15T12:49:08.032 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes_hit_set_archive": 0, 2024-05-15T12:49:08.032 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush": 0, 2024-05-15T12:49:08.033 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_kb": 0, 2024-05-15T12:49:08.033 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict": 0, 2024-05-15T12:49:08.033 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_kb": 0, 2024-05-15T12:49:08.033 INFO:tasks.workunit.client.0.smithi005.stdout: "num_promote": 0, 2024-05-15T12:49:08.033 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_mode_high": 0, 2024-05-15T12:49:08.033 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_mode_low": 0, 2024-05-15T12:49:08.033 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_mode_some": 0, 2024-05-15T12:49:08.033 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_mode_full": 0, 2024-05-15T12:49:08.033 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_pinned": 0, 2024-05-15T12:49:08.033 INFO:tasks.workunit.client.0.smithi005.stdout: "num_legacy_snapsets": 0, 2024-05-15T12:49:08.033 INFO:tasks.workunit.client.0.smithi005.stdout: "num_large_omap_objects": 0, 2024-05-15T12:49:08.033 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_manifest": 0, 2024-05-15T12:49:08.033 INFO:tasks.workunit.client.0.smithi005.stdout: "num_omap_bytes": 0, 2024-05-15T12:49:08.033 INFO:tasks.workunit.client.0.smithi005.stdout: "num_omap_keys": 0, 2024-05-15T12:49:08.033 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_repaired": 0 2024-05-15T12:49:08.033 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.034 INFO:tasks.workunit.client.0.smithi005.stdout: "up": [ 2024-05-15T12:49:08.034 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T12:49:08.034 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T12:49:08.034 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T12:49:08.034 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T12:49:08.034 INFO:tasks.workunit.client.0.smithi005.stdout: 2147483647 2024-05-15T12:49:08.034 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.034 INFO:tasks.workunit.client.0.smithi005.stdout: "acting": [ 2024-05-15T12:49:08.034 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T12:49:08.034 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T12:49:08.034 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T12:49:08.034 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T12:49:08.035 INFO:tasks.workunit.client.0.smithi005.stdout: 2147483647 2024-05-15T12:49:08.035 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.035 INFO:tasks.workunit.client.0.smithi005.stdout: "avail_no_missing": [ 2024-05-15T12:49:08.035 INFO:tasks.workunit.client.0.smithi005.stdout: "3(0)", 2024-05-15T12:49:08.035 INFO:tasks.workunit.client.0.smithi005.stdout: "0(3)", 2024-05-15T12:49:08.035 INFO:tasks.workunit.client.0.smithi005.stdout: "1(1)", 2024-05-15T12:49:08.035 INFO:tasks.workunit.client.0.smithi005.stdout: "4(2)" 2024-05-15T12:49:08.035 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.035 INFO:tasks.workunit.client.0.smithi005.stdout: "object_location_counts": [ 2024-05-15T12:49:08.035 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T12:49:08.035 INFO:tasks.workunit.client.0.smithi005.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-05-15T12:49:08.035 INFO:tasks.workunit.client.0.smithi005.stdout: "objects": 301 2024-05-15T12:49:08.035 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T12:49:08.035 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.035 INFO:tasks.workunit.client.0.smithi005.stdout: "blocked_by": [], 2024-05-15T12:49:08.035 INFO:tasks.workunit.client.0.smithi005.stdout: "up_primary": 3, 2024-05-15T12:49:08.036 INFO:tasks.workunit.client.0.smithi005.stdout: "acting_primary": 3, 2024-05-15T12:49:08.036 INFO:tasks.workunit.client.0.smithi005.stdout: "purged_snaps": [] 2024-05-15T12:49:08.036 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.036 INFO:tasks.workunit.client.0.smithi005.stdout: "empty": 0, 2024-05-15T12:49:08.036 INFO:tasks.workunit.client.0.smithi005.stdout: "dne": 0, 2024-05-15T12:49:08.036 INFO:tasks.workunit.client.0.smithi005.stdout: "incomplete": 0, 2024-05-15T12:49:08.036 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_started": 66, 2024-05-15T12:49:08.036 INFO:tasks.workunit.client.0.smithi005.stdout: "hit_set_history": { 2024-05-15T12:49:08.036 INFO:tasks.workunit.client.0.smithi005.stdout: "current_last_update": "0'0", 2024-05-15T12:49:08.036 INFO:tasks.workunit.client.0.smithi005.stdout: "history": [] 2024-05-15T12:49:08.036 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T12:49:08.036 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.036 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T12:49:08.036 INFO:tasks.workunit.client.0.smithi005.stdout: "peer": "1(1)", 2024-05-15T12:49:08.036 INFO:tasks.workunit.client.0.smithi005.stdout: "pgid": "2.0s1", 2024-05-15T12:49:08.036 INFO:tasks.workunit.client.0.smithi005.stdout: "shared": "1", 2024-05-15T12:49:08.037 INFO:tasks.workunit.client.0.smithi005.stdout: "last_update": "56'301", 2024-05-15T12:49:08.037 INFO:tasks.workunit.client.0.smithi005.stdout: "last_complete": "56'301", 2024-05-15T12:49:08.037 INFO:tasks.workunit.client.0.smithi005.stdout: "log_tail": "56'200", 2024-05-15T12:49:08.037 INFO:tasks.workunit.client.0.smithi005.stdout: "last_user_version": 301, 2024-05-15T12:49:08.037 INFO:tasks.workunit.client.0.smithi005.stdout: "last_backfill": "MAX", 2024-05-15T12:49:08.037 INFO:tasks.workunit.client.0.smithi005.stdout: "purged_snaps": [], 2024-05-15T12:49:08.037 INFO:tasks.workunit.client.0.smithi005.stdout: "history": { 2024-05-15T12:49:08.037 INFO:tasks.workunit.client.0.smithi005.stdout: "epoch_created": 48, 2024-05-15T12:49:08.037 INFO:tasks.workunit.client.0.smithi005.stdout: "epoch_pool_created": 48, 2024-05-15T12:49:08.037 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_started": 66, 2024-05-15T12:49:08.037 INFO:tasks.workunit.client.0.smithi005.stdout: "last_interval_started": 64, 2024-05-15T12:49:08.037 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_clean": 49, 2024-05-15T12:49:08.037 INFO:tasks.workunit.client.0.smithi005.stdout: "last_interval_clean": 48, 2024-05-15T12:49:08.037 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_split": 0, 2024-05-15T12:49:08.037 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_marked_full": 0, 2024-05-15T12:49:08.037 INFO:tasks.workunit.client.0.smithi005.stdout: "same_up_since": 64, 2024-05-15T12:49:08.037 INFO:tasks.workunit.client.0.smithi005.stdout: "same_interval_since": 64, 2024-05-15T12:49:08.038 INFO:tasks.workunit.client.0.smithi005.stdout: "same_primary_since": 48, 2024-05-15T12:49:08.038 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub": "0'0", 2024-05-15T12:49:08.038 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.038 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub": "0'0", 2024-05-15T12:49:08.038 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.038 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.038 INFO:tasks.workunit.client.0.smithi005.stdout: "prior_readable_until_ub": 0 2024-05-15T12:49:08.038 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.038 INFO:tasks.workunit.client.0.smithi005.stdout: "stats": { 2024-05-15T12:49:08.038 INFO:tasks.workunit.client.0.smithi005.stdout: "version": "56'301", 2024-05-15T12:49:08.038 INFO:tasks.workunit.client.0.smithi005.stdout: "reported_seq": 332, 2024-05-15T12:49:08.038 INFO:tasks.workunit.client.0.smithi005.stdout: "reported_epoch": 56, 2024-05-15T12:49:08.038 INFO:tasks.workunit.client.0.smithi005.stdout: "state": "active+undersized+degraded", 2024-05-15T12:49:08.038 INFO:tasks.workunit.client.0.smithi005.stdout: "last_fresh": "2024-05-15T12:48:20.822012+0000", 2024-05-15T12:49:08.038 INFO:tasks.workunit.client.0.smithi005.stdout: "last_change": "2024-05-15T12:46:53.562212+0000", 2024-05-15T12:49:08.039 INFO:tasks.workunit.client.0.smithi005.stdout: "last_active": "2024-05-15T12:48:20.822012+0000", 2024-05-15T12:49:08.039 INFO:tasks.workunit.client.0.smithi005.stdout: "last_peered": "2024-05-15T12:48:20.822012+0000", 2024-05-15T12:49:08.039 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean": "2024-05-15T12:46:51.099810+0000", 2024-05-15T12:49:08.039 INFO:tasks.workunit.client.0.smithi005.stdout: "last_became_active": "2024-05-15T12:46:53.562212+0000", 2024-05-15T12:49:08.039 INFO:tasks.workunit.client.0.smithi005.stdout: "last_became_peered": "2024-05-15T12:46:53.562212+0000", 2024-05-15T12:49:08.039 INFO:tasks.workunit.client.0.smithi005.stdout: "last_unstale": "2024-05-15T12:48:20.822012+0000", 2024-05-15T12:49:08.039 INFO:tasks.workunit.client.0.smithi005.stdout: "last_undegraded": "2024-05-15T12:46:53.235098+0000", 2024-05-15T12:49:08.039 INFO:tasks.workunit.client.0.smithi005.stdout: "last_fullsized": "2024-05-15T12:46:53.234482+0000", 2024-05-15T12:49:08.039 INFO:tasks.workunit.client.0.smithi005.stdout: "mapping_epoch": 64, 2024-05-15T12:49:08.039 INFO:tasks.workunit.client.0.smithi005.stdout: "log_start": "56'200", 2024-05-15T12:49:08.039 INFO:tasks.workunit.client.0.smithi005.stdout: "ondisk_log_start": "56'200", 2024-05-15T12:49:08.039 INFO:tasks.workunit.client.0.smithi005.stdout: "created": 48, 2024-05-15T12:49:08.039 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_clean": 49, 2024-05-15T12:49:08.039 INFO:tasks.workunit.client.0.smithi005.stdout: "parent": "0.0", 2024-05-15T12:49:08.039 INFO:tasks.workunit.client.0.smithi005.stdout: "parent_split_bits": 0, 2024-05-15T12:49:08.039 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub": "0'0", 2024-05-15T12:49:08.040 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.040 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub": "0'0", 2024-05-15T12:49:08.040 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.040 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.040 INFO:tasks.workunit.client.0.smithi005.stdout: "objects_scrubbed": 0, 2024-05-15T12:49:08.040 INFO:tasks.workunit.client.0.smithi005.stdout: "log_size": 101, 2024-05-15T12:49:08.040 INFO:tasks.workunit.client.0.smithi005.stdout: "log_dups_size": 200, 2024-05-15T12:49:08.040 INFO:tasks.workunit.client.0.smithi005.stdout: "ondisk_log_size": 101, 2024-05-15T12:49:08.040 INFO:tasks.workunit.client.0.smithi005.stdout: "stats_invalid": false, 2024-05-15T12:49:08.040 INFO:tasks.workunit.client.0.smithi005.stdout: "dirty_stats_invalid": false, 2024-05-15T12:49:08.040 INFO:tasks.workunit.client.0.smithi005.stdout: "omap_stats_invalid": false, 2024-05-15T12:49:08.040 INFO:tasks.workunit.client.0.smithi005.stdout: "hitset_stats_invalid": false, 2024-05-15T12:49:08.041 INFO:tasks.workunit.client.0.smithi005.stdout: "hitset_bytes_stats_invalid": false, 2024-05-15T12:49:08.041 INFO:tasks.workunit.client.0.smithi005.stdout: "pin_stats_invalid": false, 2024-05-15T12:49:08.041 INFO:tasks.workunit.client.0.smithi005.stdout: "manifest_stats_invalid": false, 2024-05-15T12:49:08.041 INFO:tasks.workunit.client.0.smithi005.stdout: "snaptrimq_len": 0, 2024-05-15T12:49:08.041 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_duration": 0, 2024-05-15T12:49:08.041 INFO:tasks.workunit.client.0.smithi005.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-05-16T18:05:59.846940+0000", 2024-05-15T12:49:08.041 INFO:tasks.workunit.client.0.smithi005.stdout: "scrub_duration": 0, 2024-05-15T12:49:08.041 INFO:tasks.workunit.client.0.smithi005.stdout: "objects_trimmed": 0, 2024-05-15T12:49:08.041 INFO:tasks.workunit.client.0.smithi005.stdout: "snaptrim_duration": 0, 2024-05-15T12:49:08.041 INFO:tasks.workunit.client.0.smithi005.stdout: "stat_sum": { 2024-05-15T12:49:08.041 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes": 1232896, 2024-05-15T12:49:08.041 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects": 301, 2024-05-15T12:49:08.041 INFO:tasks.workunit.client.0.smithi005.stdout: "num_object_clones": 0, 2024-05-15T12:49:08.041 INFO:tasks.workunit.client.0.smithi005.stdout: "num_object_copies": 1505, 2024-05-15T12:49:08.041 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_missing_on_primary": 0, 2024-05-15T12:49:08.041 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_missing": 1, 2024-05-15T12:49:08.042 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_degraded": 301, 2024-05-15T12:49:08.042 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_misplaced": 0, 2024-05-15T12:49:08.042 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_unfound": 0, 2024-05-15T12:49:08.042 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_dirty": 301, 2024-05-15T12:49:08.042 INFO:tasks.workunit.client.0.smithi005.stdout: "num_whiteouts": 0, 2024-05-15T12:49:08.042 INFO:tasks.workunit.client.0.smithi005.stdout: "num_read": 0, 2024-05-15T12:49:08.042 INFO:tasks.workunit.client.0.smithi005.stdout: "num_read_kb": 0, 2024-05-15T12:49:08.042 INFO:tasks.workunit.client.0.smithi005.stdout: "num_write": 301, 2024-05-15T12:49:08.042 INFO:tasks.workunit.client.0.smithi005.stdout: "num_write_kb": 1204, 2024-05-15T12:49:08.042 INFO:tasks.workunit.client.0.smithi005.stdout: "num_scrub_errors": 0, 2024-05-15T12:49:08.043 INFO:tasks.workunit.client.0.smithi005.stdout: "num_shallow_scrub_errors": 0, 2024-05-15T12:49:08.043 INFO:tasks.workunit.client.0.smithi005.stdout: "num_deep_scrub_errors": 0, 2024-05-15T12:49:08.043 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_recovered": 0, 2024-05-15T12:49:08.043 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes_recovered": 0, 2024-05-15T12:49:08.043 INFO:tasks.workunit.client.0.smithi005.stdout: "num_keys_recovered": 0, 2024-05-15T12:49:08.043 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_omap": 0, 2024-05-15T12:49:08.043 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_hit_set_archive": 0, 2024-05-15T12:49:08.043 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes_hit_set_archive": 0, 2024-05-15T12:49:08.043 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush": 0, 2024-05-15T12:49:08.043 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_kb": 0, 2024-05-15T12:49:08.043 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict": 0, 2024-05-15T12:49:08.043 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_kb": 0, 2024-05-15T12:49:08.043 INFO:tasks.workunit.client.0.smithi005.stdout: "num_promote": 0, 2024-05-15T12:49:08.043 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_mode_high": 0, 2024-05-15T12:49:08.043 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_mode_low": 0, 2024-05-15T12:49:08.043 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_mode_some": 0, 2024-05-15T12:49:08.044 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_mode_full": 0, 2024-05-15T12:49:08.044 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_pinned": 0, 2024-05-15T12:49:08.044 INFO:tasks.workunit.client.0.smithi005.stdout: "num_legacy_snapsets": 0, 2024-05-15T12:49:08.044 INFO:tasks.workunit.client.0.smithi005.stdout: "num_large_omap_objects": 0, 2024-05-15T12:49:08.044 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_manifest": 0, 2024-05-15T12:49:08.044 INFO:tasks.workunit.client.0.smithi005.stdout: "num_omap_bytes": 0, 2024-05-15T12:49:08.044 INFO:tasks.workunit.client.0.smithi005.stdout: "num_omap_keys": 0, 2024-05-15T12:49:08.044 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_repaired": 0 2024-05-15T12:49:08.044 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.044 INFO:tasks.workunit.client.0.smithi005.stdout: "up": [ 2024-05-15T12:49:08.044 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T12:49:08.044 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T12:49:08.044 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T12:49:08.044 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T12:49:08.044 INFO:tasks.workunit.client.0.smithi005.stdout: 2147483647 2024-05-15T12:49:08.044 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.045 INFO:tasks.workunit.client.0.smithi005.stdout: "acting": [ 2024-05-15T12:49:08.045 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T12:49:08.045 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T12:49:08.045 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T12:49:08.045 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T12:49:08.045 INFO:tasks.workunit.client.0.smithi005.stdout: 2147483647 2024-05-15T12:49:08.045 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.045 INFO:tasks.workunit.client.0.smithi005.stdout: "avail_no_missing": [ 2024-05-15T12:49:08.045 INFO:tasks.workunit.client.0.smithi005.stdout: "3(0)", 2024-05-15T12:49:08.045 INFO:tasks.workunit.client.0.smithi005.stdout: "0(3)", 2024-05-15T12:49:08.045 INFO:tasks.workunit.client.0.smithi005.stdout: "1(1)", 2024-05-15T12:49:08.045 INFO:tasks.workunit.client.0.smithi005.stdout: "4(2)" 2024-05-15T12:49:08.045 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.045 INFO:tasks.workunit.client.0.smithi005.stdout: "object_location_counts": [ 2024-05-15T12:49:08.045 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T12:49:08.045 INFO:tasks.workunit.client.0.smithi005.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-05-15T12:49:08.046 INFO:tasks.workunit.client.0.smithi005.stdout: "objects": 301 2024-05-15T12:49:08.046 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T12:49:08.046 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.046 INFO:tasks.workunit.client.0.smithi005.stdout: "blocked_by": [], 2024-05-15T12:49:08.046 INFO:tasks.workunit.client.0.smithi005.stdout: "up_primary": 3, 2024-05-15T12:49:08.046 INFO:tasks.workunit.client.0.smithi005.stdout: "acting_primary": 3, 2024-05-15T12:49:08.046 INFO:tasks.workunit.client.0.smithi005.stdout: "purged_snaps": [] 2024-05-15T12:49:08.046 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.046 INFO:tasks.workunit.client.0.smithi005.stdout: "empty": 0, 2024-05-15T12:49:08.046 INFO:tasks.workunit.client.0.smithi005.stdout: "dne": 0, 2024-05-15T12:49:08.046 INFO:tasks.workunit.client.0.smithi005.stdout: "incomplete": 0, 2024-05-15T12:49:08.047 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_started": 66, 2024-05-15T12:49:08.047 INFO:tasks.workunit.client.0.smithi005.stdout: "hit_set_history": { 2024-05-15T12:49:08.047 INFO:tasks.workunit.client.0.smithi005.stdout: "current_last_update": "0'0", 2024-05-15T12:49:08.047 INFO:tasks.workunit.client.0.smithi005.stdout: "history": [] 2024-05-15T12:49:08.047 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T12:49:08.047 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.047 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T12:49:08.047 INFO:tasks.workunit.client.0.smithi005.stdout: "peer": "4(2)", 2024-05-15T12:49:08.047 INFO:tasks.workunit.client.0.smithi005.stdout: "pgid": "2.0s2", 2024-05-15T12:49:08.047 INFO:tasks.workunit.client.0.smithi005.stdout: "shared": "2", 2024-05-15T12:49:08.047 INFO:tasks.workunit.client.0.smithi005.stdout: "last_update": "56'301", 2024-05-15T12:49:08.047 INFO:tasks.workunit.client.0.smithi005.stdout: "last_complete": "56'301", 2024-05-15T12:49:08.047 INFO:tasks.workunit.client.0.smithi005.stdout: "log_tail": "56'200", 2024-05-15T12:49:08.047 INFO:tasks.workunit.client.0.smithi005.stdout: "last_user_version": 301, 2024-05-15T12:49:08.047 INFO:tasks.workunit.client.0.smithi005.stdout: "last_backfill": "MAX", 2024-05-15T12:49:08.047 INFO:tasks.workunit.client.0.smithi005.stdout: "purged_snaps": [], 2024-05-15T12:49:08.047 INFO:tasks.workunit.client.0.smithi005.stdout: "history": { 2024-05-15T12:49:08.048 INFO:tasks.workunit.client.0.smithi005.stdout: "epoch_created": 48, 2024-05-15T12:49:08.048 INFO:tasks.workunit.client.0.smithi005.stdout: "epoch_pool_created": 48, 2024-05-15T12:49:08.048 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_started": 66, 2024-05-15T12:49:08.048 INFO:tasks.workunit.client.0.smithi005.stdout: "last_interval_started": 64, 2024-05-15T12:49:08.048 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_clean": 49, 2024-05-15T12:49:08.048 INFO:tasks.workunit.client.0.smithi005.stdout: "last_interval_clean": 48, 2024-05-15T12:49:08.048 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_split": 0, 2024-05-15T12:49:08.048 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_marked_full": 0, 2024-05-15T12:49:08.048 INFO:tasks.workunit.client.0.smithi005.stdout: "same_up_since": 64, 2024-05-15T12:49:08.048 INFO:tasks.workunit.client.0.smithi005.stdout: "same_interval_since": 64, 2024-05-15T12:49:08.048 INFO:tasks.workunit.client.0.smithi005.stdout: "same_primary_since": 48, 2024-05-15T12:49:08.048 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub": "0'0", 2024-05-15T12:49:08.048 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.048 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub": "0'0", 2024-05-15T12:49:08.048 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.048 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.049 INFO:tasks.workunit.client.0.smithi005.stdout: "prior_readable_until_ub": 0 2024-05-15T12:49:08.049 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.049 INFO:tasks.workunit.client.0.smithi005.stdout: "stats": { 2024-05-15T12:49:08.049 INFO:tasks.workunit.client.0.smithi005.stdout: "version": "56'301", 2024-05-15T12:49:08.049 INFO:tasks.workunit.client.0.smithi005.stdout: "reported_seq": 332, 2024-05-15T12:49:08.049 INFO:tasks.workunit.client.0.smithi005.stdout: "reported_epoch": 56, 2024-05-15T12:49:08.049 INFO:tasks.workunit.client.0.smithi005.stdout: "state": "active+undersized+degraded", 2024-05-15T12:49:08.049 INFO:tasks.workunit.client.0.smithi005.stdout: "last_fresh": "2024-05-15T12:48:20.822012+0000", 2024-05-15T12:49:08.049 INFO:tasks.workunit.client.0.smithi005.stdout: "last_change": "2024-05-15T12:46:53.562212+0000", 2024-05-15T12:49:08.049 INFO:tasks.workunit.client.0.smithi005.stdout: "last_active": "2024-05-15T12:48:20.822012+0000", 2024-05-15T12:49:08.049 INFO:tasks.workunit.client.0.smithi005.stdout: "last_peered": "2024-05-15T12:48:20.822012+0000", 2024-05-15T12:49:08.049 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean": "2024-05-15T12:46:51.099810+0000", 2024-05-15T12:49:08.049 INFO:tasks.workunit.client.0.smithi005.stdout: "last_became_active": "2024-05-15T12:46:53.562212+0000", 2024-05-15T12:49:08.049 INFO:tasks.workunit.client.0.smithi005.stdout: "last_became_peered": "2024-05-15T12:46:53.562212+0000", 2024-05-15T12:49:08.049 INFO:tasks.workunit.client.0.smithi005.stdout: "last_unstale": "2024-05-15T12:48:20.822012+0000", 2024-05-15T12:49:08.050 INFO:tasks.workunit.client.0.smithi005.stdout: "last_undegraded": "2024-05-15T12:46:53.235098+0000", 2024-05-15T12:49:08.050 INFO:tasks.workunit.client.0.smithi005.stdout: "last_fullsized": "2024-05-15T12:46:53.234482+0000", 2024-05-15T12:49:08.050 INFO:tasks.workunit.client.0.smithi005.stdout: "mapping_epoch": 64, 2024-05-15T12:49:08.050 INFO:tasks.workunit.client.0.smithi005.stdout: "log_start": "56'200", 2024-05-15T12:49:08.050 INFO:tasks.workunit.client.0.smithi005.stdout: "ondisk_log_start": "56'200", 2024-05-15T12:49:08.050 INFO:tasks.workunit.client.0.smithi005.stdout: "created": 48, 2024-05-15T12:49:08.050 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_clean": 49, 2024-05-15T12:49:08.050 INFO:tasks.workunit.client.0.smithi005.stdout: "parent": "0.0", 2024-05-15T12:49:08.050 INFO:tasks.workunit.client.0.smithi005.stdout: "parent_split_bits": 0, 2024-05-15T12:49:08.050 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub": "0'0", 2024-05-15T12:49:08.050 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.050 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub": "0'0", 2024-05-15T12:49:08.051 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.051 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean_scrub_stamp": "2024-05-15T12:46:43.900220+0000", 2024-05-15T12:49:08.051 INFO:tasks.workunit.client.0.smithi005.stdout: "objects_scrubbed": 0, 2024-05-15T12:49:08.051 INFO:tasks.workunit.client.0.smithi005.stdout: "log_size": 101, 2024-05-15T12:49:08.051 INFO:tasks.workunit.client.0.smithi005.stdout: "log_dups_size": 200, 2024-05-15T12:49:08.051 INFO:tasks.workunit.client.0.smithi005.stdout: "ondisk_log_size": 101, 2024-05-15T12:49:08.051 INFO:tasks.workunit.client.0.smithi005.stdout: "stats_invalid": false, 2024-05-15T12:49:08.051 INFO:tasks.workunit.client.0.smithi005.stdout: "dirty_stats_invalid": false, 2024-05-15T12:49:08.051 INFO:tasks.workunit.client.0.smithi005.stdout: "omap_stats_invalid": false, 2024-05-15T12:49:08.051 INFO:tasks.workunit.client.0.smithi005.stdout: "hitset_stats_invalid": false, 2024-05-15T12:49:08.051 INFO:tasks.workunit.client.0.smithi005.stdout: "hitset_bytes_stats_invalid": false, 2024-05-15T12:49:08.051 INFO:tasks.workunit.client.0.smithi005.stdout: "pin_stats_invalid": false, 2024-05-15T12:49:08.052 INFO:tasks.workunit.client.0.smithi005.stdout: "manifest_stats_invalid": false, 2024-05-15T12:49:08.052 INFO:tasks.workunit.client.0.smithi005.stdout: "snaptrimq_len": 0, 2024-05-15T12:49:08.052 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_duration": 0, 2024-05-15T12:49:08.052 INFO:tasks.workunit.client.0.smithi005.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-05-16T18:05:59.846940+0000", 2024-05-15T12:49:08.052 INFO:tasks.workunit.client.0.smithi005.stdout: "scrub_duration": 0, 2024-05-15T12:49:08.052 INFO:tasks.workunit.client.0.smithi005.stdout: "objects_trimmed": 0, 2024-05-15T12:49:08.052 INFO:tasks.workunit.client.0.smithi005.stdout: "snaptrim_duration": 0, 2024-05-15T12:49:08.052 INFO:tasks.workunit.client.0.smithi005.stdout: "stat_sum": { 2024-05-15T12:49:08.052 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes": 1232896, 2024-05-15T12:49:08.052 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects": 301, 2024-05-15T12:49:08.052 INFO:tasks.workunit.client.0.smithi005.stdout: "num_object_clones": 0, 2024-05-15T12:49:08.052 INFO:tasks.workunit.client.0.smithi005.stdout: "num_object_copies": 1505, 2024-05-15T12:49:08.052 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_missing_on_primary": 0, 2024-05-15T12:49:08.053 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_missing": 0, 2024-05-15T12:49:08.053 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_degraded": 301, 2024-05-15T12:49:08.053 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_misplaced": 0, 2024-05-15T12:49:08.053 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_unfound": 0, 2024-05-15T12:49:08.053 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_dirty": 301, 2024-05-15T12:49:08.053 INFO:tasks.workunit.client.0.smithi005.stdout: "num_whiteouts": 0, 2024-05-15T12:49:08.053 INFO:tasks.workunit.client.0.smithi005.stdout: "num_read": 0, 2024-05-15T12:49:08.053 INFO:tasks.workunit.client.0.smithi005.stdout: "num_read_kb": 0, 2024-05-15T12:49:08.053 INFO:tasks.workunit.client.0.smithi005.stdout: "num_write": 301, 2024-05-15T12:49:08.053 INFO:tasks.workunit.client.0.smithi005.stdout: "num_write_kb": 1204, 2024-05-15T12:49:08.053 INFO:tasks.workunit.client.0.smithi005.stdout: "num_scrub_errors": 0, 2024-05-15T12:49:08.053 INFO:tasks.workunit.client.0.smithi005.stdout: "num_shallow_scrub_errors": 0, 2024-05-15T12:49:08.053 INFO:tasks.workunit.client.0.smithi005.stdout: "num_deep_scrub_errors": 0, 2024-05-15T12:49:08.053 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_recovered": 0, 2024-05-15T12:49:08.054 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes_recovered": 0, 2024-05-15T12:49:08.054 INFO:tasks.workunit.client.0.smithi005.stdout: "num_keys_recovered": 0, 2024-05-15T12:49:08.054 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_omap": 0, 2024-05-15T12:49:08.054 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_hit_set_archive": 0, 2024-05-15T12:49:08.054 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes_hit_set_archive": 0, 2024-05-15T12:49:08.054 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush": 0, 2024-05-15T12:49:08.054 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_kb": 0, 2024-05-15T12:49:08.054 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict": 0, 2024-05-15T12:49:08.054 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_kb": 0, 2024-05-15T12:49:08.054 INFO:tasks.workunit.client.0.smithi005.stdout: "num_promote": 0, 2024-05-15T12:49:08.054 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_mode_high": 0, 2024-05-15T12:49:08.054 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_mode_low": 0, 2024-05-15T12:49:08.054 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_mode_some": 0, 2024-05-15T12:49:08.054 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_mode_full": 0, 2024-05-15T12:49:08.054 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_pinned": 0, 2024-05-15T12:49:08.054 INFO:tasks.workunit.client.0.smithi005.stdout: "num_legacy_snapsets": 0, 2024-05-15T12:49:08.055 INFO:tasks.workunit.client.0.smithi005.stdout: "num_large_omap_objects": 0, 2024-05-15T12:49:08.055 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_manifest": 0, 2024-05-15T12:49:08.055 INFO:tasks.workunit.client.0.smithi005.stdout: "num_omap_bytes": 0, 2024-05-15T12:49:08.055 INFO:tasks.workunit.client.0.smithi005.stdout: "num_omap_keys": 0, 2024-05-15T12:49:08.055 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_repaired": 0 2024-05-15T12:49:08.055 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.055 INFO:tasks.workunit.client.0.smithi005.stdout: "up": [ 2024-05-15T12:49:08.055 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T12:49:08.055 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T12:49:08.055 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T12:49:08.055 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T12:49:08.055 INFO:tasks.workunit.client.0.smithi005.stdout: 2147483647 2024-05-15T12:49:08.055 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.055 INFO:tasks.workunit.client.0.smithi005.stdout: "acting": [ 2024-05-15T12:49:08.055 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T12:49:08.055 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T12:49:08.055 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T12:49:08.056 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T12:49:08.056 INFO:tasks.workunit.client.0.smithi005.stdout: 2147483647 2024-05-15T12:49:08.056 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.056 INFO:tasks.workunit.client.0.smithi005.stdout: "avail_no_missing": [ 2024-05-15T12:49:08.056 INFO:tasks.workunit.client.0.smithi005.stdout: "3(0)", 2024-05-15T12:49:08.056 INFO:tasks.workunit.client.0.smithi005.stdout: "0(3)", 2024-05-15T12:49:08.056 INFO:tasks.workunit.client.0.smithi005.stdout: "1(1)", 2024-05-15T12:49:08.056 INFO:tasks.workunit.client.0.smithi005.stdout: "4(2)" 2024-05-15T12:49:08.056 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.056 INFO:tasks.workunit.client.0.smithi005.stdout: "object_location_counts": [ 2024-05-15T12:49:08.056 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T12:49:08.056 INFO:tasks.workunit.client.0.smithi005.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-05-15T12:49:08.056 INFO:tasks.workunit.client.0.smithi005.stdout: "objects": 301 2024-05-15T12:49:08.056 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T12:49:08.056 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.057 INFO:tasks.workunit.client.0.smithi005.stdout: "blocked_by": [], 2024-05-15T12:49:08.057 INFO:tasks.workunit.client.0.smithi005.stdout: "up_primary": 3, 2024-05-15T12:49:08.057 INFO:tasks.workunit.client.0.smithi005.stdout: "acting_primary": 3, 2024-05-15T12:49:08.057 INFO:tasks.workunit.client.0.smithi005.stdout: "purged_snaps": [] 2024-05-15T12:49:08.057 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.057 INFO:tasks.workunit.client.0.smithi005.stdout: "empty": 0, 2024-05-15T12:49:08.057 INFO:tasks.workunit.client.0.smithi005.stdout: "dne": 0, 2024-05-15T12:49:08.057 INFO:tasks.workunit.client.0.smithi005.stdout: "incomplete": 0, 2024-05-15T12:49:08.057 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_started": 66, 2024-05-15T12:49:08.057 INFO:tasks.workunit.client.0.smithi005.stdout: "hit_set_history": { 2024-05-15T12:49:08.057 INFO:tasks.workunit.client.0.smithi005.stdout: "current_last_update": "0'0", 2024-05-15T12:49:08.057 INFO:tasks.workunit.client.0.smithi005.stdout: "history": [] 2024-05-15T12:49:08.057 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T12:49:08.057 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T12:49:08.057 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.057 INFO:tasks.workunit.client.0.smithi005.stdout: "recovery_state": [ 2024-05-15T12:49:08.058 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T12:49:08.058 INFO:tasks.workunit.client.0.smithi005.stdout: "name": "Started/Primary/Active", 2024-05-15T12:49:08.058 INFO:tasks.workunit.client.0.smithi005.stdout: "enter_time": "2024-05-15T12:49:03.244994+0000", 2024-05-15T12:49:08.058 INFO:tasks.workunit.client.0.smithi005.stdout: "might_have_unfound": [ 2024-05-15T12:49:08.058 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T12:49:08.058 INFO:tasks.workunit.client.0.smithi005.stdout: "osd": "0(3)", 2024-05-15T12:49:08.058 INFO:tasks.workunit.client.0.smithi005.stdout: "status": "already probed" 2024-05-15T12:49:08.058 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.058 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T12:49:08.058 INFO:tasks.workunit.client.0.smithi005.stdout: "osd": "1(1)", 2024-05-15T12:49:08.058 INFO:tasks.workunit.client.0.smithi005.stdout: "status": "already probed" 2024-05-15T12:49:08.058 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.058 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T12:49:08.058 INFO:tasks.workunit.client.0.smithi005.stdout: "osd": "2(4)", 2024-05-15T12:49:08.058 INFO:tasks.workunit.client.0.smithi005.stdout: "status": "osd is down" 2024-05-15T12:49:08.058 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.059 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T12:49:08.059 INFO:tasks.workunit.client.0.smithi005.stdout: "osd": "4(2)", 2024-05-15T12:49:08.059 INFO:tasks.workunit.client.0.smithi005.stdout: "status": "already probed" 2024-05-15T12:49:08.059 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T12:49:08.059 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.059 INFO:tasks.workunit.client.0.smithi005.stdout: "recovery_progress": { 2024-05-15T12:49:08.059 INFO:tasks.workunit.client.0.smithi005.stdout: "backfill_targets": [], 2024-05-15T12:49:08.059 INFO:tasks.workunit.client.0.smithi005.stdout: "waiting_on_backfill": [], 2024-05-15T12:49:08.059 INFO:tasks.workunit.client.0.smithi005.stdout: "last_backfill_started": "MIN", 2024-05-15T12:49:08.059 INFO:tasks.workunit.client.0.smithi005.stdout: "backfill_info": { 2024-05-15T12:49:08.059 INFO:tasks.workunit.client.0.smithi005.stdout: "begin": "MIN", 2024-05-15T12:49:08.059 INFO:tasks.workunit.client.0.smithi005.stdout: "end": "MIN", 2024-05-15T12:49:08.059 INFO:tasks.workunit.client.0.smithi005.stdout: "objects": [] 2024-05-15T12:49:08.059 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.059 INFO:tasks.workunit.client.0.smithi005.stdout: "peer_backfill_info": [], 2024-05-15T12:49:08.059 INFO:tasks.workunit.client.0.smithi005.stdout: "backfills_in_flight": [], 2024-05-15T12:49:08.059 INFO:tasks.workunit.client.0.smithi005.stdout: "recovering": [], 2024-05-15T12:49:08.060 INFO:tasks.workunit.client.0.smithi005.stdout: "pg_backend": { 2024-05-15T12:49:08.060 INFO:tasks.workunit.client.0.smithi005.stdout: "recovery_ops": [], 2024-05-15T12:49:08.060 INFO:tasks.workunit.client.0.smithi005.stdout: "read_ops": [] 2024-05-15T12:49:08.060 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T12:49:08.060 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T12:49:08.060 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.060 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T12:49:08.060 INFO:tasks.workunit.client.0.smithi005.stdout: "name": "Started", 2024-05-15T12:49:08.060 INFO:tasks.workunit.client.0.smithi005.stdout: "enter_time": "2024-05-15T12:49:02.401784+0000" 2024-05-15T12:49:08.060 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T12:49:08.060 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T12:49:08.060 INFO:tasks.workunit.client.0.smithi005.stdout: "scrubber": { 2024-05-15T12:49:08.060 INFO:tasks.workunit.client.0.smithi005.stdout: "active": false, 2024-05-15T12:49:08.060 INFO:tasks.workunit.client.0.smithi005.stdout: "must_scrub": false, 2024-05-15T12:49:08.060 INFO:tasks.workunit.client.0.smithi005.stdout: "must_deep_scrub": false, 2024-05-15T12:49:08.060 INFO:tasks.workunit.client.0.smithi005.stdout: "must_repair": false, 2024-05-15T12:49:08.061 INFO:tasks.workunit.client.0.smithi005.stdout: "need_auto": false, 2024-05-15T12:49:08.061 INFO:tasks.workunit.client.0.smithi005.stdout: "scrub_reg_stamp": "2024-05-16T18:05:59.846940+0000", 2024-05-15T12:49:08.061 INFO:tasks.workunit.client.0.smithi005.stdout: "schedule": "no scrub is scheduled" 2024-05-15T12:49:08.061 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T12:49:08.061 INFO:tasks.workunit.client.0.smithi005.stdout: "agent_state": {} 2024-05-15T12:49:08.061 INFO:tasks.workunit.client.0.smithi005.stdout:} 2024-05-15T12:49:08.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: grep -q obj250 2024-05-15T12:49:08.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-05-15T12:49:08.194 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-05-15T12:49:08.195 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: jq .available_might_have_unfound 2024-05-15T12:49:08.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: check=true 2024-05-15T12:49:08.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: test true == true 2024-05-15T12:49:08.355 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-05-15T12:49:08.355 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].status' 2024-05-15T12:49:08.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: eval 'check="osd' is 'down"' 2024-05-15T12:49:08.505 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: check='osd is down' 2024-05-15T12:49:08.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:576: TEST_ec_backfill_unfound: test 'osd is down' == 'osd is down' 2024-05-15T12:49:08.505 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-05-15T12:49:08.505 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].osd' 2024-05-15T12:49:08.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: eval 'check="2(4)"' 2024-05-15T12:49:08.659 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: check='2(4)' 2024-05-15T12:49:08.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: test '2(4)' == '2(4)' 2024-05-15T12:49:08.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:581: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2024-05-15T12:49:08.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-15T12:49:08.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-15T12:49:08.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-05-15T12:49:08.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-15T12:49:08.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-05-15T12:49:08.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:49:08.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:49:08.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:49:08.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:49:08.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-15T12:49:08.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-15T12:49:08.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-15T12:49:08.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-15T12:49:08.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:49:08.661 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-15T12:49:08.661 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:49:08.661 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:49:08.661 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:49:08.661 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:49:08.661 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:49:08.661 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:49:08.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:49:08.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:49:08.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:49:08.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:49:08.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:49:08.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:49:08.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:49:08.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:49:08.662 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-15T12:49:08.662 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-15T12:49:08.662 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-05-15T12:49:08.662 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-05-15T12:49:08.662 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.2 2024-05-15T12:49:08.663 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:49:08.663 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-05-15T12:49:08.663 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-05-15T12:49:08.663 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-15T12:49:08.663 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-15T12:49:08.663 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-15T12:49:08.685 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:49:08.693+0000 7f486dc347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:49:08.687 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:49:08.697+0000 7f486dc347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:49:08.688 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:49:08.697+0000 7f486dc347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:49:09.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-05-15T12:49:09.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:49:09.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-15T12:49:09.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:49:09.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:49:09.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:49:09.109 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:49:09.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:49:09.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:49:09.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:49:09.239 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:49:09.249+0000 7f486dc347c0 -1 Falling back to public interface 2024-05-15T12:49:09.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:49:10.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:49:10.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:49:10.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:49:10.535 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:49:10.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:49:10.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:49:10.954 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:49:11.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:49:11.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:49:11.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:49:11.956 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:49:11.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:49:11.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:49:12.247 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:49:12.257+0000 7f486dc347c0 -1 osd.2 63 log_to_monitors true 2024-05-15T12:49:12.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:49:13.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:49:13.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:49:13.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:49:13.399 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:49:13.400 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:49:13.400 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:49:14.086 INFO:tasks.workunit.client.0.smithi005.stdout:osd.2 up in weight 1 up_from 68 up_thru 21 down_at 64 last_clean_interval [58,63) [v2:127.0.0.1:6818/2964760133,v1:127.0.0.1:6819/2964760133] [v2:127.0.0.1:6820/2964760133,v1:127.0.0.1:6821/2964760133] exists,up 6a695c3c-f227-4356-a06e-53f491f74662 2024-05-15T12:49:14.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:49:14.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:49:14.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:49:14.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:584: TEST_ec_backfill_unfound: timeout 5 rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2024-05-15T12:49:19.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:585: TEST_ec_backfill_unfound: test 124 = 124 2024-05-15T12:49:19.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:587: TEST_ec_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-05-15T12:49:19.385 INFO:tasks.workunit.client.0.smithi005.stderr:pg has no unfound objects 2024-05-15T12:49:19.410 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:589: TEST_ec_backfill_unfound: wait_for_clean 2024-05-15T12:49:19.410 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T12:49:19.410 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T12:49:19.410 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T12:49:19.410 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T12:49:19.411 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T12:49:19.411 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T12:49:19.411 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T12:49:19.412 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T12:49:19.412 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T12:49:19.539 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T12:49:19.539 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T12:49:19.539 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T12:49:19.539 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T12:49:19.539 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T12:49:19.540 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T12:49:20.065 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T12:49:20.065 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:49:20.065 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T12:49:20.066 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:49:20.066 INFO:tasks.workunit.client.0.smithi005.stderr:4' 2024-05-15T12:49:20.066 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T12:49:20.066 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:49:20.066 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T12:49:20.278 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803851 2024-05-15T12:49:20.278 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803851 2024-05-15T12:49:20.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851' 2024-05-15T12:49:20.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:49:20.279 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T12:49:20.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574914 2024-05-15T12:49:20.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574914 2024-05-15T12:49:20.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851 1-55834574914' 2024-05-15T12:49:20.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:49:20.535 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T12:49:20.697 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776132 2024-05-15T12:49:20.697 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776132 2024-05-15T12:49:20.697 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851 1-55834574914 2-292057776132' 2024-05-15T12:49:20.697 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:49:20.697 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T12:49:20.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018933 2024-05-15T12:49:20.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018933 2024-05-15T12:49:20.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851 1-55834574914 2-292057776132 3-128849018933' 2024-05-15T12:49:20.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:49:20.852 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T12:49:21.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659176 2024-05-15T12:49:21.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659176 2024-05-15T12:49:21.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851 1-55834574914 2-292057776132 3-128849018933 4-176093659176' 2024-05-15T12:49:21.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:49:21.016 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803851 2024-05-15T12:49:21.016 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:49:21.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T12:49:21.018 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803851 2024-05-15T12:49:21.018 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:49:21.019 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803851 2024-05-15T12:49:21.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803851' 2024-05-15T12:49:21.020 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 25769803851 2024-05-15T12:49:21.020 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T12:49:21.512 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803851 -lt 25769803851 2024-05-15T12:49:21.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:49:21.513 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574914 2024-05-15T12:49:21.513 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:49:21.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T12:49:21.518 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574914 2024-05-15T12:49:21.518 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:49:21.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574914 2024-05-15T12:49:21.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574914' 2024-05-15T12:49:21.519 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 55834574914 2024-05-15T12:49:21.519 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T12:49:22.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574914 -lt 55834574914 2024-05-15T12:49:22.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:49:22.063 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-292057776132 2024-05-15T12:49:22.063 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:49:22.065 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T12:49:22.066 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-292057776132 2024-05-15T12:49:22.066 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:49:22.067 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776132 2024-05-15T12:49:22.067 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 292057776132' 2024-05-15T12:49:22.068 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 292057776132 2024-05-15T12:49:22.068 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T12:49:22.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776132 -lt 292057776132 2024-05-15T12:49:22.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:49:22.576 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-128849018933 2024-05-15T12:49:22.576 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:49:22.578 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T12:49:22.579 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-128849018933 2024-05-15T12:49:22.579 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:49:22.581 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018933 2024-05-15T12:49:22.581 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 128849018933 2024-05-15T12:49:22.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 128849018933' 2024-05-15T12:49:22.582 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T12:49:23.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018933 -lt 128849018933 2024-05-15T12:49:23.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:49:23.234 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-176093659176 2024-05-15T12:49:23.234 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:49:23.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T12:49:23.236 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-176093659176 2024-05-15T12:49:23.236 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:49:23.238 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659176 2024-05-15T12:49:23.238 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.4 seq 176093659176 2024-05-15T12:49:23.238 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 176093659176' 2024-05-15T12:49:23.238 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T12:49:23.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659176 -lt 176093659176 2024-05-15T12:49:23.661 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T12:49:23.661 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:49:23.662 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:49:24.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T12:49:24.551 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T12:49:24.551 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T12:49:24.551 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T12:49:24.552 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T12:49:24.552 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T12:49:24.552 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T12:49:24.552 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T12:49:24.972 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-15T12:49:24.973 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T12:49:24.973 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:49:24.973 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:49:25.539 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-05-15T12:49:25.539 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-05-15T12:49:25.539 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-05-15T12:49:25.539 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-05-15T12:49:25.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-15T12:49:25.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-05-15T12:49:25.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-15T12:49:25.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T12:49:25.641 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T12:49:25.641 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T12:49:25.641 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T12:49:25.641 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T12:49:25.642 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T12:49:25.642 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T12:49:26.053 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-15T12:49:26.053 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T12:49:26.054 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:49:26.054 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:49:26.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-05-15T12:49:26.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-05-15T12:49:26.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-05-15T12:49:26.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-05-15T12:49:26.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-05-15T12:49:26.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-05-15T12:49:26.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-05-15T12:49:26.606 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-05-15T12:49:26.606 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-05-15T12:49:27.225 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-05-15T12:49:27.226 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-05-15T12:49:27.226 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-05-15T12:49:27.226 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-15T12:49:27.226 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-05-15T12:49:27.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-15T12:49:27.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T12:49:27.428 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T12:49:27.428 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T12:49:27.428 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T12:49:27.428 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T12:49:27.429 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T12:49:27.429 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T12:49:27.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T12:49:27.855 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T12:49:27.856 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:49:27.856 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:49:28.400 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T12:49:28.401 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T12:49:28.401 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T12:49:28.401 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: seq 1 300 2024-05-15T12:49:28.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:28.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj1 = obj250 ']' 2024-05-15T12:49:28.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2024-05-15T12:49:28.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:28.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:28.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj2 = obj250 ']' 2024-05-15T12:49:28.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2024-05-15T12:49:28.469 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:28.470 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:28.470 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj3 = obj250 ']' 2024-05-15T12:49:28.470 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2024-05-15T12:49:28.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:28.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:28.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj4 = obj250 ']' 2024-05-15T12:49:28.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2024-05-15T12:49:28.553 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:28.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:28.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj5 = obj250 ']' 2024-05-15T12:49:28.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2024-05-15T12:49:28.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:28.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:28.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj6 = obj250 ']' 2024-05-15T12:49:28.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2024-05-15T12:49:28.637 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:28.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:28.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj7 = obj250 ']' 2024-05-15T12:49:28.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2024-05-15T12:49:28.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:28.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:28.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj8 = obj250 ']' 2024-05-15T12:49:28.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2024-05-15T12:49:28.697 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:28.698 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:28.698 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj9 = obj250 ']' 2024-05-15T12:49:28.698 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2024-05-15T12:49:28.726 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:28.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:28.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj10 = obj250 ']' 2024-05-15T12:49:28.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2024-05-15T12:49:28.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:28.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:28.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj11 = obj250 ']' 2024-05-15T12:49:28.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2024-05-15T12:49:28.785 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:28.786 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:28.786 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj12 = obj250 ']' 2024-05-15T12:49:28.786 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2024-05-15T12:49:28.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:28.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:28.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj13 = obj250 ']' 2024-05-15T12:49:28.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2024-05-15T12:49:28.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:28.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:28.845 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj14 = obj250 ']' 2024-05-15T12:49:28.845 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2024-05-15T12:49:28.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:28.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:28.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj15 = obj250 ']' 2024-05-15T12:49:28.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2024-05-15T12:49:28.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:28.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:28.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj16 = obj250 ']' 2024-05-15T12:49:28.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2024-05-15T12:49:28.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:28.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:28.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj17 = obj250 ']' 2024-05-15T12:49:28.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2024-05-15T12:49:28.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:28.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:28.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj18 = obj250 ']' 2024-05-15T12:49:28.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.010 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.010 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj19 = obj250 ']' 2024-05-15T12:49:29.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj20 = obj250 ']' 2024-05-15T12:49:29.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.068 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.069 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.069 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj21 = obj250 ']' 2024-05-15T12:49:29.069 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.097 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj22 = obj250 ']' 2024-05-15T12:49:29.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.139 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.139 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj23 = obj250 ']' 2024-05-15T12:49:29.139 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.167 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.168 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.168 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj24 = obj250 ']' 2024-05-15T12:49:29.168 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.223 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj25 = obj250 ']' 2024-05-15T12:49:29.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.253 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj26 = obj250 ']' 2024-05-15T12:49:29.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj27 = obj250 ']' 2024-05-15T12:49:29.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj28 = obj250 ']' 2024-05-15T12:49:29.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.341 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.341 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj29 = obj250 ']' 2024-05-15T12:49:29.341 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj30 = obj250 ']' 2024-05-15T12:49:29.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.398 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj31 = obj250 ']' 2024-05-15T12:49:29.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj32 = obj250 ']' 2024-05-15T12:49:29.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.456 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.457 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.458 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj33 = obj250 ']' 2024-05-15T12:49:29.458 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj34 = obj250 ']' 2024-05-15T12:49:29.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.514 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj35 = obj250 ']' 2024-05-15T12:49:29.516 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.544 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj36 = obj250 ']' 2024-05-15T12:49:29.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.573 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.574 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.574 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj37 = obj250 ']' 2024-05-15T12:49:29.574 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj38 = obj250 ']' 2024-05-15T12:49:29.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj39 = obj250 ']' 2024-05-15T12:49:29.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.662 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj40 = obj250 ']' 2024-05-15T12:49:29.662 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj41 = obj250 ']' 2024-05-15T12:49:29.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.719 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.719 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj42 = obj250 ']' 2024-05-15T12:49:29.719 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.747 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj43 = obj250 ']' 2024-05-15T12:49:29.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.777 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj44 = obj250 ']' 2024-05-15T12:49:29.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.806 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.807 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.807 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj45 = obj250 ']' 2024-05-15T12:49:29.807 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.835 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.836 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.836 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj46 = obj250 ']' 2024-05-15T12:49:29.836 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.864 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.865 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.865 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj47 = obj250 ']' 2024-05-15T12:49:29.865 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.893 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.894 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.894 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj48 = obj250 ']' 2024-05-15T12:49:29.894 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj49 = obj250 ']' 2024-05-15T12:49:29.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.952 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.952 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj50 = obj250 ']' 2024-05-15T12:49:29.953 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2024-05-15T12:49:29.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:29.982 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:29.982 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj51 = obj250 ']' 2024-05-15T12:49:29.982 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.010 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj52 = obj250 ']' 2024-05-15T12:49:30.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj53 = obj250 ']' 2024-05-15T12:49:30.041 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.069 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.070 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.070 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj54 = obj250 ']' 2024-05-15T12:49:30.070 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj55 = obj250 ']' 2024-05-15T12:49:30.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.127 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj56 = obj250 ']' 2024-05-15T12:49:30.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.156 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.157 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.157 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj57 = obj250 ']' 2024-05-15T12:49:30.157 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj58 = obj250 ']' 2024-05-15T12:49:30.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.216 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj59 = obj250 ']' 2024-05-15T12:49:30.216 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.245 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj60 = obj250 ']' 2024-05-15T12:49:30.245 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.274 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj61 = obj250 ']' 2024-05-15T12:49:30.274 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.302 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.303 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.303 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj62 = obj250 ']' 2024-05-15T12:49:30.303 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj63 = obj250 ']' 2024-05-15T12:49:30.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj64 = obj250 ']' 2024-05-15T12:49:30.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj65 = obj250 ']' 2024-05-15T12:49:30.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.418 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.419 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.419 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj66 = obj250 ']' 2024-05-15T12:49:30.419 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj67 = obj250 ']' 2024-05-15T12:49:30.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.478 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.478 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj68 = obj250 ']' 2024-05-15T12:49:30.478 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj69 = obj250 ']' 2024-05-15T12:49:30.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj70 = obj250 ']' 2024-05-15T12:49:30.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj71 = obj250 ']' 2024-05-15T12:49:30.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj72 = obj250 ']' 2024-05-15T12:49:30.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj73 = obj250 ']' 2024-05-15T12:49:30.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.653 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.653 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj74 = obj250 ']' 2024-05-15T12:49:30.653 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.682 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.682 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj75 = obj250 ']' 2024-05-15T12:49:30.682 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj76 = obj250 ']' 2024-05-15T12:49:30.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj77 = obj250 ']' 2024-05-15T12:49:30.740 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj78 = obj250 ']' 2024-05-15T12:49:30.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj79 = obj250 ']' 2024-05-15T12:49:30.798 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj80 = obj250 ']' 2024-05-15T12:49:30.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.856 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.857 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.857 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj81 = obj250 ']' 2024-05-15T12:49:30.857 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.885 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.886 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.886 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj82 = obj250 ']' 2024-05-15T12:49:30.886 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj83 = obj250 ']' 2024-05-15T12:49:30.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.968 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.968 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj84 = obj250 ']' 2024-05-15T12:49:30.968 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2024-05-15T12:49:30.996 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:30.997 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:30.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj85 = obj250 ']' 2024-05-15T12:49:30.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj86 = obj250 ']' 2024-05-15T12:49:31.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj87 = obj250 ']' 2024-05-15T12:49:31.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.106 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj88 = obj250 ']' 2024-05-15T12:49:31.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.136 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.136 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj89 = obj250 ']' 2024-05-15T12:49:31.136 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj90 = obj250 ']' 2024-05-15T12:49:31.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.212 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.212 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj91 = obj250 ']' 2024-05-15T12:49:31.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj92 = obj250 ']' 2024-05-15T12:49:31.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj93 = obj250 ']' 2024-05-15T12:49:31.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.300 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.300 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj94 = obj250 ']' 2024-05-15T12:49:31.300 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj95 = obj250 ']' 2024-05-15T12:49:31.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj96 = obj250 ']' 2024-05-15T12:49:31.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.386 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj97 = obj250 ']' 2024-05-15T12:49:31.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj98 = obj250 ']' 2024-05-15T12:49:31.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj99 = obj250 ']' 2024-05-15T12:49:31.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.474 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.475 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.475 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj100 = obj250 ']' 2024-05-15T12:49:31.475 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj101 = obj250 ']' 2024-05-15T12:49:31.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj101 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj102 = obj250 ']' 2024-05-15T12:49:31.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj102 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.561 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.562 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.562 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj103 = obj250 ']' 2024-05-15T12:49:31.562 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj103 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj104 = obj250 ']' 2024-05-15T12:49:31.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj104 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.620 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.620 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj105 = obj250 ']' 2024-05-15T12:49:31.620 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj105 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.648 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.648 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj106 = obj250 ']' 2024-05-15T12:49:31.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj106 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.677 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.677 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.678 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj107 = obj250 ']' 2024-05-15T12:49:31.678 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj107 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj108 = obj250 ']' 2024-05-15T12:49:31.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj108 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.736 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.736 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj109 = obj250 ']' 2024-05-15T12:49:31.736 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj109 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj110 = obj250 ']' 2024-05-15T12:49:31.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj110 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.793 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj111 = obj250 ']' 2024-05-15T12:49:31.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj111 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj112 = obj250 ']' 2024-05-15T12:49:31.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj112 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj113 = obj250 ']' 2024-05-15T12:49:31.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj113 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj114 = obj250 ']' 2024-05-15T12:49:31.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj114 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.909 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.910 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.910 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj115 = obj250 ']' 2024-05-15T12:49:31.910 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj115 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj116 = obj250 ']' 2024-05-15T12:49:31.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj116 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.968 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj117 = obj250 ']' 2024-05-15T12:49:31.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj117 td/test-erasure-eio/CHECK 2024-05-15T12:49:31.997 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:31.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:31.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj118 = obj250 ']' 2024-05-15T12:49:31.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj118 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj119 = obj250 ']' 2024-05-15T12:49:32.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj119 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj120 = obj250 ']' 2024-05-15T12:49:32.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj120 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj121 = obj250 ']' 2024-05-15T12:49:32.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj121 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj122 = obj250 ']' 2024-05-15T12:49:32.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj122 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.143 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj123 = obj250 ']' 2024-05-15T12:49:32.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj123 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj124 = obj250 ']' 2024-05-15T12:49:32.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj124 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.202 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.202 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj125 = obj250 ']' 2024-05-15T12:49:32.202 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj125 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.230 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.232 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.232 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj126 = obj250 ']' 2024-05-15T12:49:32.232 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj126 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj127 = obj250 ']' 2024-05-15T12:49:32.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj127 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj128 = obj250 ']' 2024-05-15T12:49:32.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj128 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj129 = obj250 ']' 2024-05-15T12:49:32.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj129 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj130 = obj250 ']' 2024-05-15T12:49:32.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj130 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.376 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj131 = obj250 ']' 2024-05-15T12:49:32.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj131 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.406 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.406 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj132 = obj250 ']' 2024-05-15T12:49:32.406 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj132 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj133 = obj250 ']' 2024-05-15T12:49:32.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj133 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj134 = obj250 ']' 2024-05-15T12:49:32.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj134 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.493 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj135 = obj250 ']' 2024-05-15T12:49:32.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj135 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj136 = obj250 ']' 2024-05-15T12:49:32.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj136 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.552 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.553 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.553 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj137 = obj250 ']' 2024-05-15T12:49:32.553 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj137 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.581 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj138 = obj250 ']' 2024-05-15T12:49:32.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj138 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj139 = obj250 ']' 2024-05-15T12:49:32.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj139 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj140 = obj250 ']' 2024-05-15T12:49:32.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj140 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj141 = obj250 ']' 2024-05-15T12:49:32.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj141 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.699 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj142 = obj250 ']' 2024-05-15T12:49:32.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj142 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj143 = obj250 ']' 2024-05-15T12:49:32.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj143 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj144 = obj250 ']' 2024-05-15T12:49:32.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj144 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.787 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.788 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.788 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj145 = obj250 ']' 2024-05-15T12:49:32.788 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj145 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.817 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj146 = obj250 ']' 2024-05-15T12:49:32.817 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj146 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.845 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.846 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.846 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj147 = obj250 ']' 2024-05-15T12:49:32.846 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj147 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj148 = obj250 ']' 2024-05-15T12:49:32.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj148 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj149 = obj250 ']' 2024-05-15T12:49:32.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj149 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj150 = obj250 ']' 2024-05-15T12:49:32.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj150 td/test-erasure-eio/CHECK 2024-05-15T12:49:32.991 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:32.992 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:32.992 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj151 = obj250 ']' 2024-05-15T12:49:32.992 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj151 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.021 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.021 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj152 = obj250 ']' 2024-05-15T12:49:33.021 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj152 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.049 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj153 = obj250 ']' 2024-05-15T12:49:33.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj153 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj154 = obj250 ']' 2024-05-15T12:49:33.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj154 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.136 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.136 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.136 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj155 = obj250 ']' 2024-05-15T12:49:33.137 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj155 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj156 = obj250 ']' 2024-05-15T12:49:33.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj156 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj157 = obj250 ']' 2024-05-15T12:49:33.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj157 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj158 = obj250 ']' 2024-05-15T12:49:33.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj158 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj159 = obj250 ']' 2024-05-15T12:49:33.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj159 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.300 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.301 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.301 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj160 = obj250 ']' 2024-05-15T12:49:33.301 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj160 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj161 = obj250 ']' 2024-05-15T12:49:33.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj161 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj162 = obj250 ']' 2024-05-15T12:49:33.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj162 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj163 = obj250 ']' 2024-05-15T12:49:33.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj163 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj164 = obj250 ']' 2024-05-15T12:49:33.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj164 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj165 = obj250 ']' 2024-05-15T12:49:33.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj165 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.530 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.530 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj166 = obj250 ']' 2024-05-15T12:49:33.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj166 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.559 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj167 = obj250 ']' 2024-05-15T12:49:33.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj167 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.589 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj168 = obj250 ']' 2024-05-15T12:49:33.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj168 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj169 = obj250 ']' 2024-05-15T12:49:33.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj169 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.647 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.648 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.648 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj170 = obj250 ']' 2024-05-15T12:49:33.648 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj170 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.677 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.677 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj171 = obj250 ']' 2024-05-15T12:49:33.677 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj171 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj172 = obj250 ']' 2024-05-15T12:49:33.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj172 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj173 = obj250 ']' 2024-05-15T12:49:33.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj173 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj174 = obj250 ']' 2024-05-15T12:49:33.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj174 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.792 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.793 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.793 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj175 = obj250 ']' 2024-05-15T12:49:33.793 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj175 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj176 = obj250 ']' 2024-05-15T12:49:33.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj176 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj177 = obj250 ']' 2024-05-15T12:49:33.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj177 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj178 = obj250 ']' 2024-05-15T12:49:33.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj178 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.908 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.909 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.909 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj179 = obj250 ']' 2024-05-15T12:49:33.909 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj179 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj180 = obj250 ']' 2024-05-15T12:49:33.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj180 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj181 = obj250 ']' 2024-05-15T12:49:33.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj181 td/test-erasure-eio/CHECK 2024-05-15T12:49:33.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:33.996 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:33.996 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj182 = obj250 ']' 2024-05-15T12:49:33.996 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj182 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.024 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.025 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.025 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj183 = obj250 ']' 2024-05-15T12:49:34.025 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj183 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.053 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj184 = obj250 ']' 2024-05-15T12:49:34.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj184 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj185 = obj250 ']' 2024-05-15T12:49:34.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj185 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.111 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj186 = obj250 ']' 2024-05-15T12:49:34.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj186 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.140 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.141 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.141 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj187 = obj250 ']' 2024-05-15T12:49:34.141 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj187 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj188 = obj250 ']' 2024-05-15T12:49:34.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj188 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.197 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.198 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj189 = obj250 ']' 2024-05-15T12:49:34.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj189 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.226 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.227 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.227 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj190 = obj250 ']' 2024-05-15T12:49:34.227 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj190 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj191 = obj250 ']' 2024-05-15T12:49:34.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj191 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj192 = obj250 ']' 2024-05-15T12:49:34.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj192 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj193 = obj250 ']' 2024-05-15T12:49:34.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj193 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj194 = obj250 ']' 2024-05-15T12:49:34.345 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj194 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.372 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.373 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.373 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj195 = obj250 ']' 2024-05-15T12:49:34.373 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj195 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.401 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj196 = obj250 ']' 2024-05-15T12:49:34.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj196 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.430 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj197 = obj250 ']' 2024-05-15T12:49:34.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj197 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.460 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj198 = obj250 ']' 2024-05-15T12:49:34.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj198 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.490 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj199 = obj250 ']' 2024-05-15T12:49:34.490 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj199 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj200 = obj250 ']' 2024-05-15T12:49:34.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj200 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj201 = obj250 ']' 2024-05-15T12:49:34.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj201 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.576 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.576 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj202 = obj250 ']' 2024-05-15T12:49:34.576 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj202 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj203 = obj250 ']' 2024-05-15T12:49:34.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj203 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj204 = obj250 ']' 2024-05-15T12:49:34.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj204 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.663 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj205 = obj250 ']' 2024-05-15T12:49:34.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj205 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.693 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.693 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj206 = obj250 ']' 2024-05-15T12:49:34.693 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj206 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.722 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.723 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.723 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj207 = obj250 ']' 2024-05-15T12:49:34.723 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj207 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.751 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.752 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.752 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj208 = obj250 ']' 2024-05-15T12:49:34.753 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj208 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.780 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.781 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.781 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj209 = obj250 ']' 2024-05-15T12:49:34.781 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj209 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.810 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.810 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj210 = obj250 ']' 2024-05-15T12:49:34.810 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj210 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.838 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.839 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.839 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj211 = obj250 ']' 2024-05-15T12:49:34.839 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj211 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.867 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.868 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.868 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj212 = obj250 ']' 2024-05-15T12:49:34.868 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj212 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.897 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.897 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.897 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj213 = obj250 ']' 2024-05-15T12:49:34.897 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj213 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.936 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj214 = obj250 ']' 2024-05-15T12:49:34.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj214 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.965 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj215 = obj250 ']' 2024-05-15T12:49:34.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj215 td/test-erasure-eio/CHECK 2024-05-15T12:49:34.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:34.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:34.996 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj216 = obj250 ']' 2024-05-15T12:49:34.996 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj216 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.024 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.025 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.025 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj217 = obj250 ']' 2024-05-15T12:49:35.025 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj217 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.053 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj218 = obj250 ']' 2024-05-15T12:49:35.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj218 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj219 = obj250 ']' 2024-05-15T12:49:35.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj219 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.136 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.137 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.137 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj220 = obj250 ']' 2024-05-15T12:49:35.137 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj220 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj221 = obj250 ']' 2024-05-15T12:49:35.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj221 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.222 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.223 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.223 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj222 = obj250 ']' 2024-05-15T12:49:35.223 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj222 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj223 = obj250 ']' 2024-05-15T12:49:35.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj223 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj224 = obj250 ']' 2024-05-15T12:49:35.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj224 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj225 = obj250 ']' 2024-05-15T12:49:35.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj225 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj226 = obj250 ']' 2024-05-15T12:49:35.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj226 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.368 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.368 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj227 = obj250 ']' 2024-05-15T12:49:35.368 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj227 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj228 = obj250 ']' 2024-05-15T12:49:35.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj228 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj229 = obj250 ']' 2024-05-15T12:49:35.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj229 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.455 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.455 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj230 = obj250 ']' 2024-05-15T12:49:35.455 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj230 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj231 = obj250 ']' 2024-05-15T12:49:35.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj231 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj232 = obj250 ']' 2024-05-15T12:49:35.514 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj232 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj233 = obj250 ']' 2024-05-15T12:49:35.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj233 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.572 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.572 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj234 = obj250 ']' 2024-05-15T12:49:35.572 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj234 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj235 = obj250 ']' 2024-05-15T12:49:35.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj235 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.629 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj236 = obj250 ']' 2024-05-15T12:49:35.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj236 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj237 = obj250 ']' 2024-05-15T12:49:35.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj237 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.687 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.688 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.688 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj238 = obj250 ']' 2024-05-15T12:49:35.688 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj238 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.716 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.717 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.717 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj239 = obj250 ']' 2024-05-15T12:49:35.717 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj239 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj240 = obj250 ']' 2024-05-15T12:49:35.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj240 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.776 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.776 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj241 = obj250 ']' 2024-05-15T12:49:35.776 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj241 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.804 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj242 = obj250 ']' 2024-05-15T12:49:35.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj242 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.833 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj243 = obj250 ']' 2024-05-15T12:49:35.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj243 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.862 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj244 = obj250 ']' 2024-05-15T12:49:35.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj244 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj245 = obj250 ']' 2024-05-15T12:49:35.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj245 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj246 = obj250 ']' 2024-05-15T12:49:35.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj246 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj247 = obj250 ']' 2024-05-15T12:49:35.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj247 td/test-erasure-eio/CHECK 2024-05-15T12:49:35.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:35.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:35.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj248 = obj250 ']' 2024-05-15T12:49:35.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj248 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj249 = obj250 ']' 2024-05-15T12:49:36.010 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj249 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.038 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj250 = obj250 ']' 2024-05-15T12:49:36.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.065 INFO:tasks.workunit.client.0.smithi005.stderr:error getting pool-jerasure/obj250: (2) No such file or directory 2024-05-15T12:49:36.067 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.067 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj251 = obj250 ']' 2024-05-15T12:49:36.067 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj251 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.096 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.096 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj252 = obj250 ']' 2024-05-15T12:49:36.096 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj252 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj253 = obj250 ']' 2024-05-15T12:49:36.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj253 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.154 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj254 = obj250 ']' 2024-05-15T12:49:36.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj254 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj255 = obj250 ']' 2024-05-15T12:49:36.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj255 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj256 = obj250 ']' 2024-05-15T12:49:36.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj256 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj257 = obj250 ']' 2024-05-15T12:49:36.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj257 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj258 = obj250 ']' 2024-05-15T12:49:36.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj258 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.300 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.301 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.301 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj259 = obj250 ']' 2024-05-15T12:49:36.301 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj259 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj260 = obj250 ']' 2024-05-15T12:49:36.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj260 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj261 = obj250 ']' 2024-05-15T12:49:36.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj261 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.389 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj262 = obj250 ']' 2024-05-15T12:49:36.389 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj262 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.417 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.417 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.418 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj263 = obj250 ']' 2024-05-15T12:49:36.418 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj263 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj264 = obj250 ']' 2024-05-15T12:49:36.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj264 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.474 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.475 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.475 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj265 = obj250 ']' 2024-05-15T12:49:36.475 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj265 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj266 = obj250 ']' 2024-05-15T12:49:36.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj266 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj267 = obj250 ']' 2024-05-15T12:49:36.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj267 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.562 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.563 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.563 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj268 = obj250 ']' 2024-05-15T12:49:36.563 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj268 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj269 = obj250 ']' 2024-05-15T12:49:36.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj269 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.621 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.621 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.621 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj270 = obj250 ']' 2024-05-15T12:49:36.621 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj270 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.650 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj271 = obj250 ']' 2024-05-15T12:49:36.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj271 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.679 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.680 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.680 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj272 = obj250 ']' 2024-05-15T12:49:36.680 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj272 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj273 = obj250 ']' 2024-05-15T12:49:36.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj273 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj274 = obj250 ']' 2024-05-15T12:49:36.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj274 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj275 = obj250 ']' 2024-05-15T12:49:36.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj275 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj276 = obj250 ']' 2024-05-15T12:49:36.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj276 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.825 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj277 = obj250 ']' 2024-05-15T12:49:36.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj277 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj278 = obj250 ']' 2024-05-15T12:49:36.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj278 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj279 = obj250 ']' 2024-05-15T12:49:36.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj279 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj280 = obj250 ']' 2024-05-15T12:49:36.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj280 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj281 = obj250 ']' 2024-05-15T12:49:36.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj281 td/test-erasure-eio/CHECK 2024-05-15T12:49:36.992 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:36.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:36.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj282 = obj250 ']' 2024-05-15T12:49:36.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj282 td/test-erasure-eio/CHECK 2024-05-15T12:49:37.021 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:37.022 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:37.022 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj283 = obj250 ']' 2024-05-15T12:49:37.022 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj283 td/test-erasure-eio/CHECK 2024-05-15T12:49:37.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:37.051 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:37.051 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj284 = obj250 ']' 2024-05-15T12:49:37.051 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj284 td/test-erasure-eio/CHECK 2024-05-15T12:49:37.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:37.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:37.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj285 = obj250 ']' 2024-05-15T12:49:37.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj285 td/test-erasure-eio/CHECK 2024-05-15T12:49:37.137 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:37.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:37.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj286 = obj250 ']' 2024-05-15T12:49:37.139 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj286 td/test-erasure-eio/CHECK 2024-05-15T12:49:37.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:37.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:37.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj287 = obj250 ']' 2024-05-15T12:49:37.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj287 td/test-erasure-eio/CHECK 2024-05-15T12:49:37.223 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:37.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:37.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj288 = obj250 ']' 2024-05-15T12:49:37.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj288 td/test-erasure-eio/CHECK 2024-05-15T12:49:37.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:37.253 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:37.253 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj289 = obj250 ']' 2024-05-15T12:49:37.253 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj289 td/test-erasure-eio/CHECK 2024-05-15T12:49:37.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:37.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:37.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj290 = obj250 ']' 2024-05-15T12:49:37.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj290 td/test-erasure-eio/CHECK 2024-05-15T12:49:37.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:37.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:37.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj291 = obj250 ']' 2024-05-15T12:49:37.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj291 td/test-erasure-eio/CHECK 2024-05-15T12:49:37.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:37.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:37.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj292 = obj250 ']' 2024-05-15T12:49:37.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj292 td/test-erasure-eio/CHECK 2024-05-15T12:49:37.368 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:37.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:37.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj293 = obj250 ']' 2024-05-15T12:49:37.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj293 td/test-erasure-eio/CHECK 2024-05-15T12:49:37.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:37.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:37.398 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj294 = obj250 ']' 2024-05-15T12:49:37.398 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj294 td/test-erasure-eio/CHECK 2024-05-15T12:49:37.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:37.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:37.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj295 = obj250 ']' 2024-05-15T12:49:37.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj295 td/test-erasure-eio/CHECK 2024-05-15T12:49:37.455 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:37.456 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:37.456 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj296 = obj250 ']' 2024-05-15T12:49:37.456 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj296 td/test-erasure-eio/CHECK 2024-05-15T12:49:37.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:37.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:37.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj297 = obj250 ']' 2024-05-15T12:49:37.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj297 td/test-erasure-eio/CHECK 2024-05-15T12:49:37.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:37.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:37.514 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj298 = obj250 ']' 2024-05-15T12:49:37.514 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj298 td/test-erasure-eio/CHECK 2024-05-15T12:49:37.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:37.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:37.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj299 = obj250 ']' 2024-05-15T12:49:37.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj299 td/test-erasure-eio/CHECK 2024-05-15T12:49:37.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:37.572 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-15T12:49:37.572 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj300 = obj250 ']' 2024-05-15T12:49:37.572 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj300 td/test-erasure-eio/CHECK 2024-05-15T12:49:37.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:37.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:602: TEST_ec_backfill_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T12:49:37.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:604: TEST_ec_backfill_unfound: delete_erasure_coded_pool pool-jerasure 2024-05-15T12:49:37.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-15T12:49:37.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-15T12:49:38.137 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-jerasure' does not exist 2024-05-15T12:49:38.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-15T12:49:39.269 INFO:tasks.workunit.client.0.smithi005.stderr:erasure-code-profile myprofile does not exist 2024-05-15T12:49:39.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-15T12:49:39.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-15T12:49:39.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:49:39.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-15T12:49:39.291 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:49:39.291 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:49:39.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:49:39.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:49:39.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:49:39.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:49:39.604 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:49:39.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:49:39.606 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:49:39.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:49:39.608 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:49:39.608 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:49:39.609 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:49:39.609 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:49:39.610 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:49:39.610 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:49:39.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:49:39.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:49:39.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-15T12:49:39.683 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:49:39.683 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:49:39.683 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:49:39.683 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80702 2024-05-15T12:49:39.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:49:39.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:49:39.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-15T12:49:39.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-15T12:49:39.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-15T12:49:39.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-15T12:49:39.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-15T12:49:39.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:49:39.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-15T12:49:39.686 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:49:39.686 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:49:39.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:49:39.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:49:39.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:49:39.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:49:39.689 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:49:39.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:49:39.691 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:49:39.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:49:39.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:49:39.692 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:49:39.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:49:39.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:49:39.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:49:39.694 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:49:39.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:49:39.697 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:49:39.697 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-15T12:49:39.699 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:49:39.699 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:49:39.699 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:49:39.699 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80702 2024-05-15T12:49:39.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:49:39.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:49:39.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-15T12:49:39.702 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T12:49:39.702 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:49:39.703 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:49:39.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.80702 2024-05-15T12:49:39.704 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T12:49:39.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-15T12:49:39.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T12:49:39.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-15T12:49:39.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-15T12:49:39.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-15T12:49:39.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T12:49:39.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T12:49:39.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T12:49:39.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-15T12:49:39.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-15T12:49:40.112 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T12:49:40.112 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:49:40.112 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:49:40.113 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:49:40.113 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:49:40.113 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:49:40.113 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:49:40.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-15T12:49:40.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T12:49:40.358 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T12:49:40.358 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:49:40.358 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:49:40.358 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T12:49:40.359 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T12:49:40.359 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:49:40.359 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:49:40.359 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:49:40.360 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:49:40.360 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:49:40.360 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:49:40.360 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T12:49:40.360 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:49:40.360 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.80702/ceph-mon.a.asok config get fsid 2024-05-15T12:49:40.478 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T12:49:40.479 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:49:40.479 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:49:40.479 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T12:49:40.479 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T12:49:40.479 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:49:40.479 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:49:40.479 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:49:40.480 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:49:40.480 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:49:40.480 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:49:40.480 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T12:49:40.480 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:49:40.480 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.80702/ceph-mon.a.asok config get mon_host 2024-05-15T12:49:40.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-15T12:49:40.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-15T12:49:40.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T12:49:40.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T12:49:40.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T12:49:40.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-15T12:49:40.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T12:49:41.507 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T12:49:41.507 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:49:41.507 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:49:41.507 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:49:41.507 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:49:41.507 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:49:41.507 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:49:41.508 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:49:41.509 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:49:41.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-15T12:49:41.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-15T12:49:41.868 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd' already exists 2024-05-15T12:49:41.885 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T12:49:42.887 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-15T12:49:42.887 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:49:42.888 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:49:42.888 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:49:42.888 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:49:42.888 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:49:42.888 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T12:49:42.889 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-15T12:49:42.889 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-mon.a.asok log flush 2024-05-15T12:49:42.991 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:49:43.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-15T12:49:43.006 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:49:40.168+0000 7f88cf3a1c80 0 load: jerasure load: lrc 2024-05-15T12:49:43.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_object_attr_read_error td/test-erasure-eio 2024-05-15T12:49:43.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:359: TEST_ec_object_attr_read_error: local dir=td/test-erasure-eio 2024-05-15T12:49:43.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:360: TEST_ec_object_attr_read_error: local objname=myobject 2024-05-15T12:49:43.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:362: TEST_ec_object_attr_read_error: setup_osds 7 2024-05-15T12:49:43.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-05-15T12:49:43.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-15T12:49:43.008 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-05-15T12:49:43.010 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-05-15T12:49:43.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:49:43.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-15T12:49:43.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:49:43.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:49:43.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T12:49:43.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:49:43.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-15T12:49:43.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:49:43.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:49:43.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:49:43.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:49:43.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-15T12:49:43.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-15T12:49:43.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:49:43.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:49:43.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:49:43.013 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:49:43.013 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:49:43.013 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:49:43.013 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:49:43.013 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:49:43.013 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:49:43.015 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:49:43.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:49:43.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:49:43.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:49:43.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:49:43.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:49:43.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:49:43.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:49:43.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:49:43.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:49:43.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:49:43.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:49:43.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:49:43.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-15T12:49:43.016 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:49:43.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e819e8bf-6f45-40f7-869e-13e103a2daef 2024-05-15T12:49:43.016 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 e819e8bf-6f45-40f7-869e-13e103a2daef 2024-05-15T12:49:43.017 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e819e8bf-6f45-40f7-869e-13e103a2daef' 2024-05-15T12:49:43.017 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:49:43.033 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBnr0Rma8K3AhAAjwhldabKbgwQgbar8f1kEg== 2024-05-15T12:49:43.033 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBnr0Rma8K3AhAAjwhldabKbgwQgbar8f1kEg=="}' 2024-05-15T12:49:43.033 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e819e8bf-6f45-40f7-869e-13e103a2daef -i td/test-erasure-eio/0/new.json 2024-05-15T12:49:43.336 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:49:43.351 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-15T12:49:43.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBnr0Rma8K3AhAAjwhldabKbgwQgbar8f1kEg== --osd-uuid e819e8bf-6f45-40f7-869e-13e103a2daef 2024-05-15T12:49:43.393 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:49:43.400+0000 7ffa33cd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:49:43.404 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:49:43.412+0000 7ffa33cd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:49:43.406 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:49:43.412+0000 7ffa33cd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:49:43.406 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:49:43.412+0000 7ffa33cd87c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T12:49:43.406 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:49:43.412+0000 7ffa33cd87c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T12:49:43.406 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:49:43.412+0000 7ffa33cd87c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T12:49:43.436 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:49:43.444+0000 7ffa33cd87c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-15T12:49:49.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-15T12:49:49.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:49:49.982 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T12:49:49.982 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T12:49:49.982 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:49:50.541 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T12:49:50.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T12:49:50.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:49:50.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:49:50.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:49:50.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:49:50.574 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:49:50.584+0000 7f93fecde7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:49:50.576 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:49:50.584+0000 7f93fecde7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:49:50.578 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:49:50.584+0000 7f93fecde7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:49:51.038 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T12:49:51.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:49:51.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T12:49:51.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:49:51.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:49:51.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:49:51.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:49:51.039 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:49:51.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:49:51.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:49:51.140 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:49:51.148+0000 7f93fecde7c0 -1 Falling back to public interface 2024-05-15T12:49:51.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:49:52.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:49:52.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:49:52.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:49:52.463 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:49:52.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:49:52.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:49:52.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:49:53.779 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:49:53.792+0000 7f93fecde7c0 -1 osd.0 0 log_to_monitors true 2024-05-15T12:49:53.935 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:49:53.936 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:49:53.936 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:49:53.936 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:49:53.936 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:49:53.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:49:54.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:49:55.508 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:49:55.509 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:49:55.509 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:49:55.509 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:49:55.510 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:49:55.510 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:49:56.017 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:49:57.019 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:49:57.019 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:49:57.020 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:49:57.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:49:57.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:49:57.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:49:57.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:49:58.441 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:49:58.442 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:49:58.442 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:49:58.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:49:58.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:49:58.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:49:58.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:49:59.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:49:59.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:49:59.970 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:49:59.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:49:59.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:49:59.971 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:50:00.441 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:01.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:01.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:01.443 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:50:01.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:50:01.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:01.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:50:01.915 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:02.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:02.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:02.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:50:02.917 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:50:02.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:02.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:50:03.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:04.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:04.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:04.365 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:50:04.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:50:04.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:04.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:50:04.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:05.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:05.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:05.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:50:05.925 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:50:05.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:05.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:50:06.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:07.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:07.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:07.382 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:50:07.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:50:07.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:07.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:50:08.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:09.239 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:09.239 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:09.239 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:50:09.240 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:50:09.240 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:09.240 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:50:10.094 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:11.096 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:11.096 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:11.096 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:50:11.097 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:50:11.097 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:11.097 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:50:12.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:13.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:13.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:13.184 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:50:13.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:50:13.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:13.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:50:13.838 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:14.839 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:14.840 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:14.840 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:50:14.840 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:50:14.841 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:14.841 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:50:15.479 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:16.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:16.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:16.481 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:50:16.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:50:16.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:16.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:50:17.617 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:18.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:18.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:18.619 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:50:18.620 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:50:18.620 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:18.620 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:50:19.175 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2393981290,v1:127.0.0.1:6803/2393981290] [v2:127.0.0.1:6804/2393981290,v1:127.0.0.1:6805/2393981290] exists,up e819e8bf-6f45-40f7-869e-13e103a2daef 2024-05-15T12:50:19.177 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:50:19.177 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:50:19.177 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:50:19.177 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:50:19.177 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-15T12:50:19.177 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:50:19.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:50:19.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-15T12:50:19.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:50:19.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-15T12:50:19.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:50:19.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:50:19.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:50:19.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:50:19.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-15T12:50:19.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-15T12:50:19.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:50:19.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:50:19.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:50:19.178 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:50:19.179 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:50:19.179 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:50:19.179 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:50:19.179 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:50:19.179 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:50:19.180 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:50:19.180 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:50:19.180 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:50:19.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:50:19.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:50:19.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:50:19.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:50:19.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:50:19.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:50:19.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:50:19.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:50:19.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:50:19.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:50:19.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-15T12:50:19.181 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:50:19.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=11db2660-8c98-4791-9811-dfa5eec1be18 2024-05-15T12:50:19.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 11db2660-8c98-4791-9811-dfa5eec1be18' 2024-05-15T12:50:19.182 INFO:tasks.workunit.client.0.smithi005.stdout:add osd1 11db2660-8c98-4791-9811-dfa5eec1be18 2024-05-15T12:50:19.182 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:50:19.205 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCLr0RmjMP2DBAAwC70L+51dCyah7QA9RlmWA== 2024-05-15T12:50:19.205 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCLr0RmjMP2DBAAwC70L+51dCyah7QA9RlmWA=="}' 2024-05-15T12:50:19.205 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 11db2660-8c98-4791-9811-dfa5eec1be18 -i td/test-erasure-eio/1/new.json 2024-05-15T12:50:19.829 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:50:19.847 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-15T12:50:19.848 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCLr0RmjMP2DBAAwC70L+51dCyah7QA9RlmWA== --osd-uuid 11db2660-8c98-4791-9811-dfa5eec1be18 2024-05-15T12:50:19.884 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:50:19.895+0000 7f55405d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:50:19.898 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:50:19.907+0000 7f55405d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:50:19.899 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:50:19.911+0000 7f55405d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:50:19.900 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:50:19.911+0000 7f55405d27c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T12:50:19.900 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:50:19.911+0000 7f55405d27c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T12:50:19.900 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:50:19.911+0000 7f55405d27c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T12:50:19.937 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:50:19.947+0000 7f55405d27c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-15T12:50:26.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-15T12:50:26.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:50:26.289 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd1 key to auth repository 2024-05-15T12:50:26.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-15T12:50:26.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:50:26.913 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.1 2024-05-15T12:50:26.914 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-15T12:50:26.914 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:50:26.914 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:50:26.914 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:50:26.914 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:50:26.948 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:50:26.959+0000 7f4401bc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:50:26.950 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:50:26.963+0000 7f4401bc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:50:26.951 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:50:26.963+0000 7f4401bc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:50:27.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-15T12:50:27.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:50:27.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-15T12:50:27.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:50:27.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:50:27.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:27.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:50:27.367 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:50:27.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:27.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:50:27.535 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:50:27.547+0000 7f4401bc17c0 -1 Falling back to public interface 2024-05-15T12:50:27.802 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:28.804 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:28.804 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:50:28.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:28.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:50:28.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:28.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:50:29.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:30.195 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:50:30.207+0000 7f4401bc17c0 -1 osd.1 0 log_to_monitors true 2024-05-15T12:50:30.231 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:30.231 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:30.232 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:50:30.232 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:50:30.232 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:50:30.232 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:30.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:31.983 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:31.984 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:31.984 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:50:31.984 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:50:31.984 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:31.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:50:32.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:33.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:33.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:33.411 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:50:33.412 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:50:33.412 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:33.412 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:50:33.841 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:34.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:34.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:34.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:50:34.844 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:50:34.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:34.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:50:35.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:36.379 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:36.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:36.380 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:50:36.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:50:36.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:36.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:50:36.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:37.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:37.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:37.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:50:37.940 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:50:37.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:37.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:50:38.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:39.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:39.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:39.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:50:39.411 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:50:39.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:39.412 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:50:39.878 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:40.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:40.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:40.880 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:50:40.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:50:40.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:40.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:50:41.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:42.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:42.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:42.399 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:50:42.400 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:50:42.400 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:42.400 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:50:42.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:43.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:43.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:43.860 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:50:43.861 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:50:43.861 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:43.861 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:50:44.385 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:45.386 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:45.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:45.387 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:50:45.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:50:45.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:45.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:50:46.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:47.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:47.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:47.084 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:50:47.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:50:47.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:47.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:50:48.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:49.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:49.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:49.007 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:50:49.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:50:49.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:49.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:50:50.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:51.196 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:51.196 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:51.196 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:50:51.197 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:50:51.197 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:51.197 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:50:51.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:52.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:52.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:52.758 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:50:52.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:50:52.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:52.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:50:53.597 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:54.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:54.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:54.600 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:50:54.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:50:54.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:54.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:50:55.221 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:50:56.221 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:50:56.221 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:50:56.221 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T12:50:56.222 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T12:50:56.222 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:50:56.222 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:50:56.810 INFO:tasks.workunit.client.0.smithi005.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3585555414,v1:127.0.0.1:6811/3585555414] [v2:127.0.0.1:6812/3585555414,v1:127.0.0.1:6813/3585555414] exists,up 11db2660-8c98-4791-9811-dfa5eec1be18 2024-05-15T12:50:56.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:50:56.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:50:56.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:50:56.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:50:56.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-15T12:50:56.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:50:56.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:50:56.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-15T12:50:56.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:50:56.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-15T12:50:56.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:50:56.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:50:56.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:50:56.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:50:56.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-15T12:50:56.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-15T12:50:56.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:50:56.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:50:56.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:50:56.814 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:50:56.814 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:50:56.815 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:50:56.815 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:50:56.815 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:50:56.815 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:50:56.815 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:50:56.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:50:56.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:50:56.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:50:56.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:50:56.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:50:56.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:50:56.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:50:56.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:50:56.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:50:56.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:50:56.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:50:56.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:50:56.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-15T12:50:56.816 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:50:56.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3352110d-6724-4458-bff8-d71308f42453 2024-05-15T12:50:56.817 INFO:tasks.workunit.client.0.smithi005.stdout:add osd2 3352110d-6724-4458-bff8-d71308f42453 2024-05-15T12:50:56.817 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 3352110d-6724-4458-bff8-d71308f42453' 2024-05-15T12:50:56.817 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:50:56.838 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCwr0RmrJywMhAAeG+43OHC1clP7Xd46p3Rwg== 2024-05-15T12:50:56.838 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCwr0RmrJywMhAAeG+43OHC1clP7Xd46p3Rwg=="}' 2024-05-15T12:50:56.838 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3352110d-6724-4458-bff8-d71308f42453 -i td/test-erasure-eio/2/new.json 2024-05-15T12:50:57.356 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:50:57.371 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-15T12:50:57.372 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCwr0RmrJywMhAAeG+43OHC1clP7Xd46p3Rwg== --osd-uuid 3352110d-6724-4458-bff8-d71308f42453 2024-05-15T12:50:57.407 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:50:57.418+0000 7f6f2857c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:50:57.439 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:50:57.450+0000 7f6f2857c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:50:57.440 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:50:57.450+0000 7f6f2857c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:50:57.441 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:50:57.450+0000 7f6f2857c7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T12:50:57.441 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:50:57.450+0000 7f6f2857c7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T12:50:57.441 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:50:57.450+0000 7f6f2857c7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T12:50:57.469 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:50:57.478+0000 7f6f2857c7c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-15T12:51:04.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-15T12:51:04.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:51:04.157 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd2 key to auth repository 2024-05-15T12:51:04.157 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-15T12:51:04.157 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:51:04.729 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.2 2024-05-15T12:51:04.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-15T12:51:04.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:51:04.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:51:04.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:51:04.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:51:04.762 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:51:04.774+0000 7f40f3a547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:51:04.764 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:51:04.774+0000 7f40f3a547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:51:04.766 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:51:04.774+0000 7f40f3a547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:51:05.152 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-15T12:51:05.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:51:05.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-15T12:51:05.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:51:05.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:51:05.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:05.177 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:51:05.177 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:51:05.177 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:51:05.177 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:05.332 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:51:05.342+0000 7f40f3a547c0 -1 Falling back to public interface 2024-05-15T12:51:05.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:06.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:06.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:06.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:51:06.569 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:51:06.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:06.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:51:06.997 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:07.735 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:51:07.746+0000 7f40f3a547c0 -1 osd.2 0 log_to_monitors true 2024-05-15T12:51:07.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:07.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:07.999 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:51:08.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:51:08.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:08.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:51:08.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:09.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:09.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:09.607 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:51:09.608 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:51:09.608 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:09.608 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:51:10.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:11.064 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:11.064 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:11.065 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:51:11.065 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:51:11.065 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:11.065 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:51:11.598 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:12.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:12.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:12.600 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:51:12.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:51:12.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:12.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:51:13.070 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:14.072 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:14.072 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:14.072 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:51:14.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:51:14.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:14.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:51:14.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:15.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:15.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:15.673 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:51:15.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:51:15.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:15.674 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:51:16.152 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:17.154 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:17.154 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:17.154 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:51:17.154 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:51:17.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:17.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:51:17.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:18.674 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:18.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:18.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:51:18.675 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:51:18.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:18.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:51:19.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:20.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:20.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:20.166 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:51:20.166 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:51:20.166 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:20.166 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:51:20.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:21.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:21.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:21.823 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:51:21.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:51:21.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:21.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:51:22.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:23.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:23.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:23.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:51:23.352 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:51:23.353 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:23.353 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:51:24.521 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:25.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:25.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:25.523 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:51:25.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:51:25.525 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:25.525 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:51:26.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:27.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:27.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:27.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:51:27.340 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:51:27.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:27.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:51:28.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:29.152 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:29.152 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:29.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:51:29.153 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:51:29.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:29.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:51:29.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:30.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:30.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:30.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:51:30.995 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:51:30.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:30.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:51:31.942 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:32.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:32.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:32.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:51:32.944 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:51:32.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:32.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:51:33.391 INFO:tasks.workunit.client.0.smithi005.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2203068501,v1:127.0.0.1:6819/2203068501] [v2:127.0.0.1:6820/2203068501,v1:127.0.0.1:6821/2203068501] exists,up 3352110d-6724-4458-bff8-d71308f42453 2024-05-15T12:51:33.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:51:33.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:51:33.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:51:33.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:51:33.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-15T12:51:33.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:51:33.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:51:33.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-15T12:51:33.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:51:33.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-15T12:51:33.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:51:33.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:51:33.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:51:33.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:51:33.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-15T12:51:33.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-15T12:51:33.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:51:33.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:51:33.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:51:33.394 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:51:33.395 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:51:33.395 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:51:33.395 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:51:33.395 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:51:33.395 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:51:33.396 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:51:33.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:51:33.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:51:33.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:51:33.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:51:33.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:51:33.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:51:33.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:51:33.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:51:33.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:51:33.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:51:33.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:51:33.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:51:33.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-15T12:51:33.397 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:51:33.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=172f2d92-657b-4279-a24a-b804af59173f 2024-05-15T12:51:33.398 INFO:tasks.workunit.client.0.smithi005.stdout:add osd3 172f2d92-657b-4279-a24a-b804af59173f 2024-05-15T12:51:33.398 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 172f2d92-657b-4279-a24a-b804af59173f' 2024-05-15T12:51:33.398 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:51:33.420 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDVr0RmWGzNGRAADE1SnqQUk7usf0orbUTwXA== 2024-05-15T12:51:33.420 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDVr0RmWGzNGRAADE1SnqQUk7usf0orbUTwXA=="}' 2024-05-15T12:51:33.420 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 172f2d92-657b-4279-a24a-b804af59173f -i td/test-erasure-eio/3/new.json 2024-05-15T12:51:34.584 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:51:34.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-15T12:51:34.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDVr0RmWGzNGRAADE1SnqQUk7usf0orbUTwXA== --osd-uuid 172f2d92-657b-4279-a24a-b804af59173f 2024-05-15T12:51:34.641 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:51:34.650+0000 7f18612917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:51:34.696 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:51:34.706+0000 7f18612917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:51:34.698 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:51:34.706+0000 7f18612917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:51:34.698 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:51:34.710+0000 7f18612917c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T12:51:34.699 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:51:34.710+0000 7f18612917c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T12:51:34.699 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:51:34.710+0000 7f18612917c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T12:51:34.831 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:51:34.842+0000 7f18612917c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-15T12:51:41.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-15T12:51:41.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:51:41.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-15T12:51:41.186 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd3 key to auth repository 2024-05-15T12:51:41.187 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:51:41.741 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.3 2024-05-15T12:51:41.741 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-15T12:51:41.741 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:51:41.742 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:51:41.742 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:51:41.742 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:51:41.774 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:51:41.782+0000 7f41557307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:51:41.791 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:51:41.802+0000 7f41557307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:51:41.792 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:51:41.802+0000 7f41557307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:51:42.147 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-15T12:51:42.147 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:51:42.148 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-15T12:51:42.148 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:51:42.148 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:51:42.148 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:42.148 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:51:42.148 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:51:42.148 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:42.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:51:42.356 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:51:42.366+0000 7f41557307c0 -1 Falling back to public interface 2024-05-15T12:51:42.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:43.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:43.561 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:43.561 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:51:43.561 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:51:43.561 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:43.562 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:51:43.983 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:44.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:44.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:44.985 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:51:44.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:51:44.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:44.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:51:45.072 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:51:45.081+0000 7f41557307c0 -1 osd.3 0 log_to_monitors true 2024-05-15T12:51:45.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:46.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:46.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:46.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:51:46.427 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:51:46.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:46.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:51:47.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:48.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:48.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:48.450 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:51:48.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:51:48.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:48.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:51:48.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:49.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:49.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:49.927 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:51:49.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:51:49.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:49.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:51:50.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:51.400 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:51.400 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:51.401 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:51:51.401 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:51:51.401 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:51.401 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:51:51.901 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:52.903 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:52.903 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:52.904 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:51:52.904 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:51:52.904 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:52.904 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:51:53.341 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:54.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:54.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:54.343 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:51:54.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:51:54.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:54.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:51:54.968 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:55.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:55.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:55.970 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:51:55.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:51:55.971 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:55.971 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:51:56.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:57.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:57.392 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:57.392 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:51:57.392 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:51:57.392 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:57.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:51:57.900 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:51:58.901 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:51:58.902 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:51:58.902 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:51:58.902 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:51:58.902 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:51:58.903 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:51:59.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:00.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:00.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:00.507 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:52:00.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:52:00.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:00.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:52:01.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:02.134 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:02.134 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:02.135 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:52:02.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:52:02.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:02.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:52:02.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:03.946 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:03.946 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:03.947 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:52:03.947 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:52:03.947 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:03.947 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:52:04.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:05.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:05.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:05.639 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:52:05.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:52:05.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:05.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:52:06.807 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:07.808 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:07.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:07.809 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:52:07.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:52:07.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:07.810 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:52:08.562 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:09.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:09.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:09.565 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:52:09.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:52:09.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:09.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:52:10.123 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:11.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:11.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:11.126 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T12:52:11.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T12:52:11.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:11.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:52:11.580 INFO:tasks.workunit.client.0.smithi005.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/695805440,v1:127.0.0.1:6827/695805440] [v2:127.0.0.1:6828/695805440,v1:127.0.0.1:6829/695805440] exists,up 172f2d92-657b-4279-a24a-b804af59173f 2024-05-15T12:52:11.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:52:11.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:52:11.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:52:11.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:52:11.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-05-15T12:52:11.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:52:11.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:52:11.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-05-15T12:52:11.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:52:11.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-05-15T12:52:11.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:52:11.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:52:11.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:52:11.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:52:11.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-05-15T12:52:11.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-05-15T12:52:11.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:52:11.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:52:11.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:52:11.585 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:52:11.585 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:52:11.585 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:52:11.585 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:52:11.585 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:52:11.585 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:52:11.585 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:52:11.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:52:11.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:52:11.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:52:11.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:52:11.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:52:11.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:52:11.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:52:11.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:52:11.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:52:11.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:52:11.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:52:11.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:52:11.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-05-15T12:52:11.587 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:52:11.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=40e6ab9c-7f17-4976-a641-32aa04b6e7b6 2024-05-15T12:52:11.587 INFO:tasks.workunit.client.0.smithi005.stdout:add osd4 40e6ab9c-7f17-4976-a641-32aa04b6e7b6 2024-05-15T12:52:11.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 40e6ab9c-7f17-4976-a641-32aa04b6e7b6' 2024-05-15T12:52:11.587 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:52:11.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7r0RmK5jEJBAAX1W5AwdAvN21cjqcKy1IQw== 2024-05-15T12:52:11.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7r0RmK5jEJBAAX1W5AwdAvN21cjqcKy1IQw=="}' 2024-05-15T12:52:11.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 40e6ab9c-7f17-4976-a641-32aa04b6e7b6 -i td/test-erasure-eio/4/new.json 2024-05-15T12:52:12.168 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:52:12.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-05-15T12:52:12.187 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7r0RmK5jEJBAAX1W5AwdAvN21cjqcKy1IQw== --osd-uuid 40e6ab9c-7f17-4976-a641-32aa04b6e7b6 2024-05-15T12:52:12.217 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:12.225+0000 7fd96f5877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:52:12.220 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:12.229+0000 7fd96f5877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:52:12.221 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:12.229+0000 7fd96f5877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:52:12.222 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:12.229+0000 7fd96f5877c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T12:52:12.222 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:12.229+0000 7fd96f5877c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T12:52:12.222 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:12.229+0000 7fd96f5877c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T12:52:12.318 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:12.329+0000 7fd96f5877c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-05-15T12:52:19.771 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-05-15T12:52:19.771 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:52:19.773 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd4 key to auth repository 2024-05-15T12:52:19.773 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-05-15T12:52:19.773 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:52:20.327 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.4 2024-05-15T12:52:20.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-05-15T12:52:20.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:52:20.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:52:20.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:52:20.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:52:20.360 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:20.369+0000 7f185e3a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:52:20.362 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:20.373+0000 7f185e3a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:52:20.364 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:20.373+0000 7f185e3a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:52:20.825 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-05-15T12:52:20.825 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:52:20.825 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-05-15T12:52:20.825 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:52:20.825 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:52:20.826 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:52:20.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:20.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:52:20.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:20.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:52:20.925 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:20.933+0000 7f185e3a07c0 -1 Falling back to public interface 2024-05-15T12:52:21.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:22.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:22.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:22.284 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:52:22.284 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:52:22.284 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:22.284 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:52:22.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:23.439 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:23.449+0000 7f185e3a07c0 -1 osd.4 0 log_to_monitors true 2024-05-15T12:52:23.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:23.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:23.712 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:52:23.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:52:23.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:23.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:52:24.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:25.401 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:25.401 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:25.401 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:52:25.401 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:52:25.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:25.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:52:26.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:27.238 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:27.238 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:27.238 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:52:27.238 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:52:27.239 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:27.239 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:52:27.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:28.857 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:28.857 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:28.858 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:52:28.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:52:28.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:28.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:52:29.423 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:30.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:30.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:30.425 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:52:30.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:52:30.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:30.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:52:30.943 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:31.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:31.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:31.946 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:52:31.946 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:52:31.946 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:31.946 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:52:32.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:33.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:33.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:33.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:52:33.451 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:52:33.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:33.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:52:33.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:34.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:34.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:34.933 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:52:34.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:52:34.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:34.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:52:35.479 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:36.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:36.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:36.481 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:52:36.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:52:36.482 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:36.482 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:52:37.047 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:38.049 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:38.049 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:38.049 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:52:38.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:52:38.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:38.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:52:38.954 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:39.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:39.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:39.956 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:52:39.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:52:39.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:39.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:52:40.842 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:41.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:41.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:41.844 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:52:41.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:52:41.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:41.845 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:52:42.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:43.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:43.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:43.924 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:52:43.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:52:43.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:43.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:52:44.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:45.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:45.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:45.874 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:52:45.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:52:45.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:45.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:52:46.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:47.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:47.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:47.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:52:47.720 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:52:47.721 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:47.721 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:52:48.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:52:49.547 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:52:49.548 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:52:49.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:49.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:52:49.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:49.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:52:50.079 INFO:tasks.workunit.client.0.smithi005.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1965862026,v1:127.0.0.1:6835/1965862026] [v2:127.0.0.1:6836/1965862026,v1:127.0.0.1:6837/1965862026] exists,up 40e6ab9c-7f17-4976-a641-32aa04b6e7b6 2024-05-15T12:52:50.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:52:50.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:52:50.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:52:50.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:52:50.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-05-15T12:52:50.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:52:50.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:52:50.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-05-15T12:52:50.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:52:50.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-05-15T12:52:50.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:52:50.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:52:50.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:52:50.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:52:50.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-05-15T12:52:50.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-05-15T12:52:50.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:52:50.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:52:50.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:52:50.083 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:52:50.083 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:52:50.084 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:52:50.084 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:52:50.084 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:52:50.084 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:52:50.084 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:52:50.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:52:50.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:52:50.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:52:50.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:52:50.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:52:50.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:52:50.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:52:50.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:52:50.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:52:50.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:52:50.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:52:50.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:52:50.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-05-15T12:52:50.086 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:52:50.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=054198a3-0e04-4c82-a66e-654655aca73d 2024-05-15T12:52:50.086 INFO:tasks.workunit.client.0.smithi005.stdout:add osd5 054198a3-0e04-4c82-a66e-654655aca73d 2024-05-15T12:52:50.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 054198a3-0e04-4c82-a66e-654655aca73d' 2024-05-15T12:52:50.086 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:52:50.106 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAisERmOzESBxAA4aXA+IzVl8uYIyA1+Ca9Ng== 2024-05-15T12:52:50.106 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAisERmOzESBxAA4aXA+IzVl8uYIyA1+Ca9Ng=="}' 2024-05-15T12:52:50.106 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 054198a3-0e04-4c82-a66e-654655aca73d -i td/test-erasure-eio/5/new.json 2024-05-15T12:52:50.856 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:52:50.878 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-05-15T12:52:50.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAisERmOzESBxAA4aXA+IzVl8uYIyA1+Ca9Ng== --osd-uuid 054198a3-0e04-4c82-a66e-654655aca73d 2024-05-15T12:52:50.916 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:50.924+0000 7fc7593f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:52:50.918 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:50.928+0000 7fc7593f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:52:50.920 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:50.928+0000 7fc7593f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:52:50.920 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:50.928+0000 7fc7593f07c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-15T12:52:50.920 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:50.928+0000 7fc7593f07c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-15T12:52:50.920 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:50.928+0000 7fc7593f07c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-15T12:52:51.111 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:51.120+0000 7fc7593f07c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-05-15T12:52:58.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-05-15T12:52:58.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:52:58.318 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd5 key to auth repository 2024-05-15T12:52:58.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-05-15T12:52:58.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:52:58.935 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-05-15T12:52:58.936 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.5 2024-05-15T12:52:58.936 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:52:58.936 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:52:58.936 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:52:58.936 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:52:58.969 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:58.976+0000 7fb9a20297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:52:58.987 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:58.996+0000 7fb9a20297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:52:58.988 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:58.996+0000 7fb9a20297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:52:59.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-05-15T12:52:59.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:52:59.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-05-15T12:52:59.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:52:59.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:52:59.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:52:59.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:52:59.367 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:52:59.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:52:59.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:52:59.572 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:52:59.580+0000 7fb9a20297c0 -1 Falling back to public interface 2024-05-15T12:52:59.808 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:00.810 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:00.810 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:00.811 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:53:00.811 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:53:00.811 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:00.811 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:53:01.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:02.129 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:53:02.136+0000 7fb9a20297c0 -1 osd.5 0 log_to_monitors true 2024-05-15T12:53:02.230 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:02.230 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:02.230 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:53:02.230 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:53:02.230 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:02.231 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:53:02.968 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:03.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:03.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:03.970 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:53:03.971 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:53:03.971 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:03.971 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:53:04.725 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:05.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:05.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:05.727 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:53:05.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:53:05.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:05.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:53:06.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:07.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:07.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:07.268 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:53:07.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:53:07.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:07.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:53:07.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:08.732 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:08.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:08.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:53:08.733 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:53:08.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:08.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:53:09.371 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:10.373 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:10.374 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:10.374 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:53:10.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:53:10.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:10.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:53:11.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:12.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:12.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:12.008 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:53:12.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:53:12.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:12.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:53:12.468 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:13.470 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:13.470 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:13.471 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:53:13.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:53:13.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:13.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:53:14.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:15.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:15.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:15.188 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:53:15.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:53:15.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:15.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:53:15.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:16.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:16.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:16.627 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:53:16.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:53:16.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:16.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:53:17.528 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:18.530 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:18.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:18.531 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:53:18.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:53:18.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:18.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:53:19.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:20.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:20.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:20.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:53:20.550 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:53:20.551 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:20.551 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:53:21.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:22.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:22.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:22.333 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:53:22.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:53:22.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:22.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:53:23.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:24.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:24.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:24.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:53:24.042 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:53:24.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:24.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:53:25.118 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:26.120 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:26.120 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:26.120 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:53:26.121 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:53:26.121 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:26.121 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:53:26.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:27.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:27.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:27.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:53:27.961 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:53:27.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:27.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:53:28.509 INFO:tasks.workunit.client.0.smithi005.stdout:osd.5 up in weight 1 up_from 51 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2233547508,v1:127.0.0.1:6843/2233547508] [v2:127.0.0.1:6844/2233547508,v1:127.0.0.1:6845/2233547508] exists,up 054198a3-0e04-4c82-a66e-654655aca73d 2024-05-15T12:53:28.544 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:53:28.544 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:53:28.544 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:53:28.544 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:53:28.544 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-05-15T12:53:28.544 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:53:28.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:53:28.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-05-15T12:53:28.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:53:28.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-05-15T12:53:28.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:53:28.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:53:28.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:53:28.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:53:28.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-05-15T12:53:28.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-05-15T12:53:28.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:53:28.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:53:28.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:53:28.545 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:53:28.546 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:53:28.546 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:53:28.546 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:53:28.546 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:53:28.546 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:53:28.546 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:53:28.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:53:28.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:53:28.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:53:28.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:53:28.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:53:28.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:53:28.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:53:28.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:53:28.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:53:28.547 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:53:28.547 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:53:28.547 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:53:28.547 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-05-15T12:53:28.547 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:53:28.547 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6a7c7d9f-4ab3-4797-b8e3-f2f19f5820c7 2024-05-15T12:53:28.547 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 6a7c7d9f-4ab3-4797-b8e3-f2f19f5820c7' 2024-05-15T12:53:28.547 INFO:tasks.workunit.client.0.smithi005.stdout:add osd6 6a7c7d9f-4ab3-4797-b8e3-f2f19f5820c7 2024-05-15T12:53:28.548 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:53:28.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBIsERmfoXoIBAAwNuHdbtHP5sq3um2/vt3pQ== 2024-05-15T12:53:28.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBIsERmfoXoIBAAwNuHdbtHP5sq3um2/vt3pQ=="}' 2024-05-15T12:53:28.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6a7c7d9f-4ab3-4797-b8e3-f2f19f5820c7 -i td/test-erasure-eio/6/new.json 2024-05-15T12:53:29.206 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:53:29.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-05-15T12:53:29.225 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBIsERmfoXoIBAAwNuHdbtHP5sq3um2/vt3pQ== --osd-uuid 6a7c7d9f-4ab3-4797-b8e3-f2f19f5820c7 2024-05-15T12:53:29.260 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:53:29.272+0000 7f82f74c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:53:29.326 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:53:29.340+0000 7f82f74c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:53:29.328 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:53:29.340+0000 7f82f74c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:53:29.328 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:53:29.340+0000 7f82f74c97c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-15T12:53:29.329 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:53:29.340+0000 7f82f74c97c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-15T12:53:29.329 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:53:29.340+0000 7f82f74c97c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-15T12:53:29.462 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:53:29.476+0000 7f82f74c97c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-05-15T12:53:36.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-05-15T12:53:36.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:53:36.328 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd6 key to auth repository 2024-05-15T12:53:36.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-05-15T12:53:36.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:53:36.946 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.6 2024-05-15T12:53:36.947 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-05-15T12:53:36.947 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:53:36.947 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:53:36.947 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:53:36.947 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:53:36.980 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:53:36.992+0000 7fe6f49cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:53:36.999 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:53:37.012+0000 7fe6f49cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:53:37.001 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:53:37.012+0000 7fe6f49cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:53:37.379 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-05-15T12:53:37.379 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:53:37.379 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-05-15T12:53:37.379 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:53:37.379 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:53:37.379 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:37.380 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:53:37.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:53:37.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:53:37.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:37.564 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:53:37.575+0000 7fe6f49cf7c0 -1 Falling back to public interface 2024-05-15T12:53:37.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:38.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:38.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:38.815 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:53:38.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:53:38.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:38.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:53:39.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:40.175 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:53:40.187+0000 7fe6f49cf7c0 -1 osd.6 0 log_to_monitors true 2024-05-15T12:53:40.294 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:40.294 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:40.294 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:53:40.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:53:40.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:40.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:53:40.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:41.982 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:41.982 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:41.982 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:53:41.983 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:53:41.983 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:41.983 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:53:42.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:43.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:43.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:43.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:53:43.711 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:53:43.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:43.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:53:44.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:45.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:45.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:45.311 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:53:45.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:53:45.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:45.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:53:45.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:46.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:46.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:46.797 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:53:46.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:53:46.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:46.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:53:47.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:48.368 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:48.368 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:48.368 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:53:48.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:53:48.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:48.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:53:48.949 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:49.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:49.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:49.951 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:53:49.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:53:49.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:49.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:53:50.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:51.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:51.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:51.504 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:53:51.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:53:51.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:51.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:53:52.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:53.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:53.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:53.110 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:53:53.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:53:53.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:53.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:53:53.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:54.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:54.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:54.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:53:54.586 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:53:54.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:54.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:53:55.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:56.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:56.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:56.291 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:53:56.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:53:56.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:56.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:53:57.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:53:58.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:53:58.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:53:58.329 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:53:58.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:53:58.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:53:58.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:53:59.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:54:00.154 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:54:00.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:54:00.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:54:00.155 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:54:00.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:54:00.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:54:00.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:54:01.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:54:01.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:54:01.821 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:54:01.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:54:01.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:54:01.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:54:02.598 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:54:03.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:54:03.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:54:03.601 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:54:03.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:54:03.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:54:03.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:54:04.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:54:05.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:54:05.389 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:54:05.389 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:54:05.389 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:54:05.389 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:54:05.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:54:05.824 INFO:tasks.workunit.client.0.smithi005.stdout:osd.6 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/866677624,v1:127.0.0.1:6851/866677624] [v2:127.0.0.1:6852/866677624,v1:127.0.0.1:6853/866677624] exists,up 6a7c7d9f-4ab3-4797-b8e3-f2f19f5820c7 2024-05-15T12:54:05.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:54:05.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:54:05.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:54:05.825 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-15T12:54:05.825 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T12:54:05.825 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T12:54:05.826 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:54:05.826 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:54:05.826 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:54:05.826 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.0.asok 2024-05-15T12:54:05.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-15T12:54:05.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-osd.0.asok log flush 2024-05-15T12:54:05.938 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:54:05.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-15T12:54:05.958 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:49:51.156+0000 7f93fecde7c0 0 load: jerasure load: lrc 2024-05-15T12:54:05.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:364: TEST_ec_object_attr_read_error: local poolname=pool-jerasure 2024-05-15T12:54:05.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:365: TEST_ec_object_attr_read_error: create_erasure_coded_pool pool-jerasure 3 2 2024-05-15T12:54:05.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-15T12:54:05.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-15T12:54:05.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-05-15T12:54:05.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-15T12:54:05.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-05-15T12:54:05.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-15T12:54:05.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-05-15T12:54:06.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-15T12:54:06.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-15T12:54:08.030 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-jerasure' already exists 2024-05-15T12:54:08.047 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T12:54:09.048 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-15T12:54:09.049 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T12:54:09.049 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T12:54:09.049 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T12:54:09.049 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T12:54:09.050 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T12:54:09.050 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T12:54:09.051 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T12:54:09.051 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T12:54:09.051 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T12:54:09.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T12:54:09.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T12:54:09.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T12:54:09.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T12:54:09.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T12:54:09.179 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T12:54:09.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T12:54:09.704 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:54:09.704 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T12:54:09.704 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:54:09.704 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T12:54:09.704 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T12:54:09.704 INFO:tasks.workunit.client.0.smithi005.stderr:6' 2024-05-15T12:54:09.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T12:54:09.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:54:09.704 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T12:54:09.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803828 2024-05-15T12:54:09.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803828 2024-05-15T12:54:09.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803828' 2024-05-15T12:54:09.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:54:09.964 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T12:54:10.118 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574892 2024-05-15T12:54:10.119 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574892 2024-05-15T12:54:10.119 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803828 1-55834574892' 2024-05-15T12:54:10.119 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:54:10.119 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T12:54:10.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378668 2024-05-15T12:54:10.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378668 2024-05-15T12:54:10.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803828 1-55834574892 2-81604378668' 2024-05-15T12:54:10.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:54:10.282 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T12:54:10.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084314 2024-05-15T12:54:10.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084314 2024-05-15T12:54:10.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803828 1-55834574892 2-81604378668 3-120259084314' 2024-05-15T12:54:10.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:54:10.480 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T12:54:10.644 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691863 2024-05-15T12:54:10.644 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691863 2024-05-15T12:54:10.644 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803828 1-55834574892 2-81604378668 3-120259084314 4-171798691863' 2024-05-15T12:54:10.644 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:54:10.644 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-15T12:54:10.802 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=219043332110 2024-05-15T12:54:10.802 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 219043332110 2024-05-15T12:54:10.802 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803828 1-55834574892 2-81604378668 3-120259084314 4-171798691863 5-219043332110' 2024-05-15T12:54:10.802 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:54:10.802 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-15T12:54:10.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005067 2024-05-15T12:54:10.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005067 2024-05-15T12:54:10.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803828 1-55834574892 2-81604378668 3-120259084314 4-171798691863 5-219043332110 6-261993005067' 2024-05-15T12:54:10.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:54:10.957 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803828 2024-05-15T12:54:10.958 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:54:10.958 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T12:54:10.959 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803828 2024-05-15T12:54:10.959 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:54:10.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803828 2024-05-15T12:54:10.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803828' 2024-05-15T12:54:10.961 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 25769803828 2024-05-15T12:54:10.961 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T12:54:11.379 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803828 -lt 25769803828 2024-05-15T12:54:11.379 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:54:11.380 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574892 2024-05-15T12:54:11.380 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:54:11.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T12:54:11.382 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574892 2024-05-15T12:54:11.382 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:54:11.383 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574892 2024-05-15T12:54:11.383 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574892' 2024-05-15T12:54:11.384 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 55834574892 2024-05-15T12:54:11.384 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T12:54:11.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574892 -lt 55834574892 2024-05-15T12:54:11.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:54:11.829 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378668 2024-05-15T12:54:11.829 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:54:11.831 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T12:54:11.832 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378668 2024-05-15T12:54:11.832 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:54:11.833 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378668 2024-05-15T12:54:11.833 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378668' 2024-05-15T12:54:11.834 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 81604378668 2024-05-15T12:54:11.834 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T12:54:12.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378668 -lt 81604378668 2024-05-15T12:54:12.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:54:12.285 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084314 2024-05-15T12:54:12.285 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:54:12.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T12:54:12.288 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084314 2024-05-15T12:54:12.288 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:54:12.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084314 2024-05-15T12:54:12.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084314' 2024-05-15T12:54:12.290 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 120259084314 2024-05-15T12:54:12.290 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T12:54:12.719 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084315 -lt 120259084314 2024-05-15T12:54:12.719 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:54:12.720 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-171798691863 2024-05-15T12:54:12.720 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:54:12.722 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T12:54:12.722 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-171798691863 2024-05-15T12:54:12.722 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:54:12.724 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691863 2024-05-15T12:54:12.724 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.4 seq 171798691863 2024-05-15T12:54:12.724 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 171798691863' 2024-05-15T12:54:12.724 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T12:54:13.159 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691863 -lt 171798691863 2024-05-15T12:54:13.159 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:54:13.160 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-219043332110 2024-05-15T12:54:13.160 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:54:13.161 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-15T12:54:13.162 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:54:13.162 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-219043332110 2024-05-15T12:54:13.166 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=219043332110 2024-05-15T12:54:13.166 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.5 seq 219043332110 2024-05-15T12:54:13.167 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 219043332110' 2024-05-15T12:54:13.167 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-15T12:54:13.581 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 219043332110 -lt 219043332110 2024-05-15T12:54:13.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:54:13.582 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-261993005067 2024-05-15T12:54:13.583 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:54:13.583 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-15T12:54:13.584 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-261993005067 2024-05-15T12:54:13.584 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:54:13.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005067 2024-05-15T12:54:13.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 261993005067' 2024-05-15T12:54:13.586 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.6 seq 261993005067 2024-05-15T12:54:13.586 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-15T12:54:14.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005067 -lt 261993005067 2024-05-15T12:54:14.013 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T12:54:14.013 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:54:14.014 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:54:14.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T12:54:14.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T12:54:14.600 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T12:54:14.600 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T12:54:14.600 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T12:54:14.601 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T12:54:14.601 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T12:54:14.601 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T12:54:15.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T12:54:15.016 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T12:54:15.016 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:54:15.016 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:54:15.596 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T12:54:15.596 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T12:54:15.596 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T12:54:15.596 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_ec_object_attr_read_error: get_primary pool-jerasure myobject 2024-05-15T12:54:15.596 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=pool-jerasure 2024-05-15T12:54:15.596 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=myobject 2024-05-15T12:54:15.597 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map pool-jerasure myobject 2024-05-15T12:54:15.597 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-05-15T12:54:16.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_ec_object_attr_read_error: local primary_osd=3 2024-05-15T12:54:16.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:369: TEST_ec_object_attr_read_error: kill_daemons td/test-erasure-eio TERM osd.3 2024-05-15T12:54:16.021 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:54:16.021 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:54:16.021 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:54:16.021 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:54:16.021 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:54:16.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:54:16.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:372: TEST_ec_object_attr_read_error: rados_put td/test-erasure-eio pool-jerasure myobject 2024-05-15T12:54:16.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-15T12:54:16.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-15T12:54:16.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-05-15T12:54:16.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T12:54:16.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-15T12:54:16.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T12:54:16.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-15T12:54:16.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T12:54:16.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-15T12:54:16.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T12:54:16.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-15T12:54:16.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-05-15T12:54:17.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:375: TEST_ec_object_attr_read_error: inject_eio ec mdata pool-jerasure myobject td/test-erasure-eio 1 2024-05-15T12:54:17.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-15T12:54:17.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-15T12:54:17.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-05-15T12:54:17.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-15T12:54:17.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-15T12:54:17.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-15T12:54:17.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-05-15T12:54:17.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-15T12:54:17.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-15T12:54:17.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-15T12:54:17.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-05-15T12:54:17.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-15T12:54:17.626 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-05-15T12:54:17.626 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T12:54:17.626 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-05-15T12:54:17.626 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-05-15T12:54:17.626 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T12:54:18.056 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2147483647 2024-05-15T12:54:18.057 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T12:54:18.057 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:54:18.057 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T12:54:18.057 INFO:tasks.workunit.client.0.smithi005.stderr:6' 2024-05-15T12:54:18.057 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2147483647 5 1 0 6 2024-05-15T12:54:18.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('2147483647' '5' '1' '0' '6') 2024-05-15T12:54:18.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-15T12:54:18.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=5 2024-05-15T12:54:18.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-15T12:54:18.057 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/5/type 2024-05-15T12:54:18.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-15T12:54:18.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 5 bluestore_debug_inject_read_err true 2024-05-15T12:54:18.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-15T12:54:18.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=5 2024-05-15T12:54:18.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-15T12:54:18.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-15T12:54:18.060 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-15T12:54:18.060 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.5 2024-05-15T12:54:18.060 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-05-15T12:54:18.060 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-05-15T12:54:18.060 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:54:18.060 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:54:18.060 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:54:18.060 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.5.asok 2024-05-15T12:54:18.060 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.80702/ceph-osd.5.asok config set bluestore_debug_inject_read_err true 2024-05-15T12:54:18.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-15T12:54:18.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-15T12:54:18.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-15T12:54:18.186 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.5 2024-05-15T12:54:18.186 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-05-15T12:54:18.186 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-05-15T12:54:18.187 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:54:18.187 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:54:18.187 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:54:18.187 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.5.asok 2024-05-15T12:54:18.187 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-15T12:54:18.187 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-osd.5.asok injectmdataerr pool-jerasure myobject 1 2024-05-15T12:54:18.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:378: TEST_ec_object_attr_read_error: activate_osd td/test-erasure-eio 3 2024-05-15T12:54:18.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-15T12:54:18.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-15T12:54:18.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-05-15T12:54:18.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-15T12:54:18.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-05-15T12:54:18.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:54:18.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:54:18.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:54:18.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:54:18.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-15T12:54:18.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-15T12:54:18.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-15T12:54:18.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-15T12:54:18.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:54:18.312 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-15T12:54:18.312 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:54:18.312 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:54:18.312 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:54:18.312 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:54:18.312 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:54:18.312 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:54:18.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:54:18.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:54:18.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:54:18.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:54:18.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:54:18.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:54:18.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:54:18.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:54:18.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-15T12:54:18.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-15T12:54:18.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-05-15T12:54:18.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-05-15T12:54:18.314 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.3 2024-05-15T12:54:18.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:54:18.314 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-05-15T12:54:18.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-05-15T12:54:18.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-15T12:54:18.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-15T12:54:18.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-15T12:54:18.345 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:54:18.355+0000 7f705671d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:54:18.348 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:54:18.359+0000 7f705671d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:54:18.349 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:54:18.359+0000 7f705671d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:54:18.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-05-15T12:54:18.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:54:18.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-15T12:54:18.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:54:18.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:54:18.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:54:18.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:54:18.769 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:54:18.770 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:54:18.770 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:54:18.919 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:54:18.931+0000 7f705671d7c0 -1 Falling back to public interface 2024-05-15T12:54:19.220 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:54:20.221 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:54:20.221 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:54:20.221 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:54:20.222 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:54:20.222 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:54:20.222 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:54:20.656 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:54:21.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:54:21.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:54:21.659 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:54:21.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:54:21.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:54:21.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:54:21.854 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:54:21.867+0000 7f705671d7c0 -1 osd.3 68 log_to_monitors true 2024-05-15T12:54:22.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:54:23.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:54:23.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:54:23.109 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:54:23.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:54:23.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:54:23.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:54:23.540 INFO:tasks.workunit.client.0.smithi005.stdout:osd.3 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [28,68) [v2:127.0.0.1:6826/2705036663,v1:127.0.0.1:6827/2705036663] [v2:127.0.0.1:6828/2705036663,v1:127.0.0.1:6829/2705036663] exists,up 172f2d92-657b-4279-a24a-b804af59173f 2024-05-15T12:54:23.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:54:23.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:54:23.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:54:23.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:381: TEST_ec_object_attr_read_error: wait_for_clean 2024-05-15T12:54:23.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T12:54:23.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T12:54:23.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T12:54:23.542 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T12:54:23.543 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T12:54:23.543 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T12:54:23.543 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T12:54:23.543 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T12:54:23.543 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T12:54:23.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T12:54:23.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T12:54:23.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T12:54:23.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T12:54:23.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T12:54:23.660 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T12:54:24.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T12:54:24.144 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T12:54:24.144 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T12:54:24.144 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T12:54:24.144 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T12:54:24.144 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T12:54:24.144 INFO:tasks.workunit.client.0.smithi005.stderr:6' 2024-05-15T12:54:24.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T12:54:24.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:54:24.144 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T12:54:24.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803833 2024-05-15T12:54:24.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803833 2024-05-15T12:54:24.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803833' 2024-05-15T12:54:24.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:54:24.346 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T12:54:24.508 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574897 2024-05-15T12:54:24.509 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574897 2024-05-15T12:54:24.509 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803833 1-55834574897' 2024-05-15T12:54:24.509 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:54:24.509 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T12:54:24.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378673 2024-05-15T12:54:24.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378673 2024-05-15T12:54:24.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803833 1-55834574897 2-81604378673' 2024-05-15T12:54:24.656 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:54:24.656 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T12:54:24.799 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645315 2024-05-15T12:54:24.799 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645315 2024-05-15T12:54:24.799 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803833 1-55834574897 2-81604378673 3-309237645315' 2024-05-15T12:54:24.799 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:54:24.799 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T12:54:24.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691867 2024-05-15T12:54:24.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691867 2024-05-15T12:54:24.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803833 1-55834574897 2-81604378673 3-309237645315 4-171798691867' 2024-05-15T12:54:24.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:54:24.994 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-15T12:54:25.152 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=219043332115 2024-05-15T12:54:25.152 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 219043332115 2024-05-15T12:54:25.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803833 1-55834574897 2-81604378673 3-309237645315 4-171798691867 5-219043332115' 2024-05-15T12:54:25.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T12:54:25.153 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-15T12:54:25.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005071 2024-05-15T12:54:25.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005071 2024-05-15T12:54:25.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803833 1-55834574897 2-81604378673 3-309237645315 4-171798691867 5-219043332115 6-261993005071' 2024-05-15T12:54:25.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:54:25.305 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803833 2024-05-15T12:54:25.305 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:54:25.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T12:54:25.306 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803833 2024-05-15T12:54:25.307 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:54:25.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803833 2024-05-15T12:54:25.308 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 25769803833 2024-05-15T12:54:25.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803833' 2024-05-15T12:54:25.308 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T12:54:25.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803833 -lt 25769803833 2024-05-15T12:54:25.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:54:25.710 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574897 2024-05-15T12:54:25.710 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:54:25.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T12:54:25.712 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574897 2024-05-15T12:54:25.712 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:54:25.714 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574897 2024-05-15T12:54:25.714 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 55834574897 2024-05-15T12:54:25.714 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574897' 2024-05-15T12:54:25.715 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T12:54:26.147 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574895 -lt 55834574897 2024-05-15T12:54:26.147 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-15T12:54:27.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-15T12:54:27.150 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T12:54:27.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574898 -lt 55834574897 2024-05-15T12:54:27.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:54:27.580 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378673 2024-05-15T12:54:27.580 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:54:27.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T12:54:27.582 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378673 2024-05-15T12:54:27.583 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:54:27.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378673 2024-05-15T12:54:27.584 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 81604378673 2024-05-15T12:54:27.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378673' 2024-05-15T12:54:27.585 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T12:54:28.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378673 -lt 81604378673 2024-05-15T12:54:28.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:54:28.056 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645315 2024-05-15T12:54:28.056 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:54:28.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T12:54:28.059 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645315 2024-05-15T12:54:28.059 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:54:28.060 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 309237645315 2024-05-15T12:54:28.060 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645315 2024-05-15T12:54:28.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645315' 2024-05-15T12:54:28.061 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T12:54:28.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645315 -lt 309237645315 2024-05-15T12:54:28.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:54:28.546 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-171798691867 2024-05-15T12:54:28.547 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:54:28.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T12:54:28.550 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-171798691867 2024-05-15T12:54:28.550 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:54:28.551 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691867 2024-05-15T12:54:28.551 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 171798691867' 2024-05-15T12:54:28.551 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.4 seq 171798691867 2024-05-15T12:54:28.551 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T12:54:28.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691868 -lt 171798691867 2024-05-15T12:54:28.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:54:28.981 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-219043332115 2024-05-15T12:54:28.982 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:54:28.983 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-15T12:54:28.984 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-219043332115 2024-05-15T12:54:28.984 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:54:28.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=219043332115 2024-05-15T12:54:28.986 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.5 seq 219043332115 2024-05-15T12:54:28.987 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 219043332115' 2024-05-15T12:54:28.987 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-15T12:54:29.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 219043332115 -lt 219043332115 2024-05-15T12:54:29.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T12:54:29.429 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-261993005071 2024-05-15T12:54:29.429 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T12:54:29.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-15T12:54:29.431 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-261993005071 2024-05-15T12:54:29.432 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T12:54:29.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005071 2024-05-15T12:54:29.433 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.6 seq 261993005071 2024-05-15T12:54:29.434 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 261993005071' 2024-05-15T12:54:29.434 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-15T12:54:29.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005072 -lt 261993005071 2024-05-15T12:54:29.855 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T12:54:29.856 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:54:29.856 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:54:30.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T12:54:30.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T12:54:30.436 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T12:54:30.436 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T12:54:30.436 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T12:54:30.436 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T12:54:30.437 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T12:54:30.437 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T12:54:30.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T12:54:30.860 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T12:54:30.860 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T12:54:30.860 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T12:54:31.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T12:54:31.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T12:54:31.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T12:54:31.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:383: TEST_ec_object_attr_read_error: rados_get td/test-erasure-eio pool-jerasure myobject 2024-05-15T12:54:31.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-15T12:54:31.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-15T12:54:31.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-05-15T12:54:31.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-15T12:54:31.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-15T12:54:31.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-05-15T12:54:31.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-15T12:54:31.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-15T12:54:31.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:385: TEST_ec_object_attr_read_error: delete_erasure_coded_pool pool-jerasure 2024-05-15T12:54:31.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-15T12:54:31.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-15T12:54:32.109 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-jerasure' does not exist 2024-05-15T12:54:32.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-15T12:54:32.864 INFO:tasks.workunit.client.0.smithi005.stderr:erasure-code-profile myprofile does not exist 2024-05-15T12:54:32.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-15T12:54:32.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-15T12:54:32.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:54:32.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-15T12:54:32.881 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:54:32.881 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:54:32.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:54:32.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:54:32.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:54:33.196 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:54:33.197 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:54:33.198 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:54:33.199 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:54:33.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:54:33.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:54:33.201 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:54:33.202 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:54:33.202 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:54:33.203 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:54:33.204 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:54:33.206 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:54:33.207 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:54:33.207 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-15T12:54:33.250 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:54:33.250 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:54:33.250 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:54:33.250 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80702 2024-05-15T12:54:33.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:54:33.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:54:33.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-15T12:54:33.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-15T12:54:33.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-15T12:54:33.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-15T12:54:33.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-15T12:54:33.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T12:54:33.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-15T12:54:33.252 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T12:54:33.252 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T12:54:33.253 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T12:54:33.253 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T12:54:33.253 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T12:54:33.255 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T12:54:33.257 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T12:54:33.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T12:54:33.257 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T12:54:33.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T12:54:33.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T12:54:33.259 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T12:54:33.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:54:33.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T12:54:33.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T12:54:33.260 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:54:33.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:54:33.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T12:54:33.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-15T12:54:33.264 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T12:54:33.264 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:54:33.264 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:54:33.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80702 2024-05-15T12:54:33.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T12:54:33.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T12:54:33.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-15T12:54:33.267 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T12:54:33.267 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:54:33.267 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:54:33.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.80702 2024-05-15T12:54:33.269 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T12:54:33.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-15T12:54:33.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T12:54:33.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-15T12:54:33.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-15T12:54:33.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-15T12:54:33.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T12:54:33.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T12:54:33.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T12:54:33.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-15T12:54:33.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-15T12:54:33.690 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T12:54:33.691 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:54:33.691 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:54:33.691 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:54:33.691 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:54:33.691 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:54:33.692 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:54:33.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-15T12:54:33.910 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T12:54:33.911 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T12:54:33.911 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:54:33.911 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:54:33.911 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T12:54:33.912 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T12:54:33.912 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:54:33.912 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:54:33.912 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:54:33.913 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:54:33.913 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:54:33.913 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:54:33.913 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T12:54:33.913 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:54:33.913 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.80702/ceph-mon.a.asok config get fsid 2024-05-15T12:54:34.029 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T12:54:34.029 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T12:54:34.030 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T12:54:34.030 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T12:54:34.031 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T12:54:34.031 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T12:54:34.031 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:54:34.031 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:54:34.031 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:54:34.031 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:54:34.031 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:54:34.031 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T12:54:34.031 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T12:54:34.031 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.80702/ceph-mon.a.asok config get mon_host 2024-05-15T12:54:34.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-15T12:54:34.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-15T12:54:34.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T12:54:34.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T12:54:34.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T12:54:34.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-15T12:54:34.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T12:54:35.042 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T12:54:35.107 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:54:35.108 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:54:35.108 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:54:35.108 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:54:35.108 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:54:35.108 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:54:35.109 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:54:35.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T12:54:35.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-15T12:54:35.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-15T12:54:35.371 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd' already exists 2024-05-15T12:54:35.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T12:54:36.391 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-15T12:54:36.391 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T12:54:36.391 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T12:54:36.392 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T12:54:36.392 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:54:36.392 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:54:36.392 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T12:54:36.392 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-15T12:54:36.392 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-mon.a.asok log flush 2024-05-15T12:54:36.493 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T12:54:36.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-15T12:54:36.509 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:54:33.747+0000 7f473fddcc80 0 load: jerasure load: lrc 2024-05-15T12:54:36.510 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_errors td/test-erasure-eio 2024-05-15T12:54:36.510 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:418: TEST_ec_recovery_multiple_errors: local dir=td/test-erasure-eio 2024-05-15T12:54:36.510 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:419: TEST_ec_recovery_multiple_errors: local objname=myobject 2024-05-15T12:54:36.510 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_recovery_multiple_errors: setup_osds 9 2024-05-15T12:54:36.510 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=9 2024-05-15T12:54:36.510 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-15T12:54:36.511 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 9 - 1 2024-05-15T12:54:36.512 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 8 2024-05-15T12:54:36.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:54:36.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-15T12:54:36.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:54:36.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:54:36.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T12:54:36.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:54:36.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-15T12:54:36.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:54:36.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:54:36.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:54:36.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:54:36.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-15T12:54:36.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-15T12:54:36.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:54:36.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:54:36.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:54:36.518 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:54:36.518 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:54:36.518 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:54:36.518 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:54:36.518 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:54:36.518 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:54:36.518 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:54:36.518 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:54:36.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:54:36.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:54:36.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:54:36.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:54:36.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:54:36.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:54:36.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:54:36.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:54:36.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:54:36.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:54:36.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:54:36.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-15T12:54:36.520 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:54:36.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2e2fe95b-dfff-4017-8a3a-deaec021da8c 2024-05-15T12:54:36.520 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 2e2fe95b-dfff-4017-8a3a-deaec021da8c 2024-05-15T12:54:36.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2e2fe95b-dfff-4017-8a3a-deaec021da8c' 2024-05-15T12:54:36.521 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:54:36.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCMsERmZnrBIBAAnfmaN/nkTy0CaqxWCokS5A== 2024-05-15T12:54:36.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCMsERmZnrBIBAAnfmaN/nkTy0CaqxWCokS5A=="}' 2024-05-15T12:54:36.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2e2fe95b-dfff-4017-8a3a-deaec021da8c -i td/test-erasure-eio/0/new.json 2024-05-15T12:54:36.839 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:54:36.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-15T12:54:36.856 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCMsERmZnrBIBAAnfmaN/nkTy0CaqxWCokS5A== --osd-uuid 2e2fe95b-dfff-4017-8a3a-deaec021da8c 2024-05-15T12:54:36.896 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:54:36.907+0000 7ffa286cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:54:36.916 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:54:36.927+0000 7ffa286cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:54:36.917 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:54:36.927+0000 7ffa286cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:54:36.918 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:54:36.931+0000 7ffa286cd7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T12:54:36.918 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:54:36.931+0000 7ffa286cd7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T12:54:36.918 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:54:36.931+0000 7ffa286cd7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T12:54:36.956 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:54:36.967+0000 7ffa286cd7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-15T12:54:42.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-15T12:54:42.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:54:42.930 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T12:54:42.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T12:54:42.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:54:43.490 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T12:54:43.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T12:54:43.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:54:43.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:54:43.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:54:43.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:54:43.516 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:54:43.526+0000 7f5df3e5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:54:43.518 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:54:43.530+0000 7f5df3e5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:54:43.520 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:54:43.530+0000 7f5df3e5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:54:43.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T12:54:43.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:54:43.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T12:54:43.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:54:43.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:54:43.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:54:43.946 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:54:43.946 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:54:43.946 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:54:43.946 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:54:44.076 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:54:44.086+0000 7f5df3e5f7c0 -1 Falling back to public interface 2024-05-15T12:54:44.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:54:45.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:54:45.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:54:45.382 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:54:45.383 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:54:45.383 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:54:45.383 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:54:45.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:54:46.716 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:54:46.726+0000 7f5df3e5f7c0 -1 osd.0 0 log_to_monitors true 2024-05-15T12:54:46.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:54:46.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:54:46.852 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:54:46.853 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:54:46.853 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:54:46.853 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:54:47.419 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:54:48.421 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:54:48.421 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:54:48.421 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:54:48.421 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:54:48.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:54:48.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:54:48.840 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:54:49.842 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:54:49.842 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:54:49.843 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:54:49.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:54:49.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:54:49.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:54:50.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:54:51.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:54:51.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:54:51.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:54:51.275 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:54:51.276 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:54:51.276 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:54:51.862 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:54:52.864 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:54:52.864 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:54:52.864 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:54:52.864 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:54:52.865 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:54:52.865 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:54:53.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:54:54.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:54:54.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:54:54.335 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:54:54.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:54:54.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:54:54.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:54:54.897 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:54:55.898 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:54:55.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:54:55.899 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:54:55.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:54:55.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:54:55.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:54:56.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:54:57.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:54:57.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:54:57.347 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:54:57.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:54:57.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:54:57.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:54:57.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:54:58.878 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:54:58.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:54:58.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:54:58.879 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:54:58.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:54:58.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:54:59.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:00.434 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:00.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:00.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:55:00.435 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:55:00.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:00.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:55:01.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:02.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:02.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:02.391 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:55:02.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:55:02.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:02.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:55:03.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:04.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:04.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:04.113 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:55:04.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:55:04.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:04.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:55:04.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:05.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:05.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:05.602 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:55:05.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:55:05.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:05.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:55:06.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:07.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:07.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:07.364 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:55:07.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:55:07.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:07.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:55:08.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:09.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:09.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:09.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:55:09.007 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:55:09.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:09.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:55:09.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:10.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:10.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:10.767 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:55:10.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:55:10.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:10.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T12:55:11.409 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2804014958,v1:127.0.0.1:6803/2804014958] [v2:127.0.0.1:6804/2804014958,v1:127.0.0.1:6805/2804014958] exists,up 2e2fe95b-dfff-4017-8a3a-deaec021da8c 2024-05-15T12:55:11.413 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:55:11.413 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:55:11.413 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:55:11.413 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:55:11.413 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-15T12:55:11.413 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:55:11.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:55:11.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-15T12:55:11.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:55:11.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-15T12:55:11.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:55:11.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:55:11.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:55:11.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:55:11.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-15T12:55:11.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-15T12:55:11.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:55:11.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:55:11.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:55:11.415 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:55:11.415 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:55:11.415 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:55:11.415 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:55:11.415 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:55:11.415 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:55:11.416 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:55:11.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:55:11.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:55:11.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:55:11.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:55:11.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:55:11.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:55:11.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:55:11.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:55:11.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:55:11.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:55:11.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:55:11.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:55:11.417 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-15T12:55:11.417 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:55:11.417 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5278715e-854b-4e16-8706-edf249203c92 2024-05-15T12:55:11.418 INFO:tasks.workunit.client.0.smithi005.stdout:add osd1 5278715e-854b-4e16-8706-edf249203c92 2024-05-15T12:55:11.418 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5278715e-854b-4e16-8706-edf249203c92' 2024-05-15T12:55:11.418 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:55:11.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCvsERm+pnvGhAAu9qtIqTfajkhtxqs+rXduw== 2024-05-15T12:55:11.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCvsERm+pnvGhAAu9qtIqTfajkhtxqs+rXduw=="}' 2024-05-15T12:55:11.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5278715e-854b-4e16-8706-edf249203c92 -i td/test-erasure-eio/1/new.json 2024-05-15T12:55:12.180 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:55:12.198 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-15T12:55:12.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCvsERm+pnvGhAAu9qtIqTfajkhtxqs+rXduw== --osd-uuid 5278715e-854b-4e16-8706-edf249203c92 2024-05-15T12:55:12.235 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:12.246+0000 7f10b941d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:55:12.246 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:12.258+0000 7f10b941d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:55:12.247 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:12.258+0000 7f10b941d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:55:12.248 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:12.258+0000 7f10b941d7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T12:55:12.248 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:12.258+0000 7f10b941d7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T12:55:12.248 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:12.258+0000 7f10b941d7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T12:55:12.286 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:12.298+0000 7f10b941d7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-15T12:55:18.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-15T12:55:18.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:55:18.728 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd1 key to auth repository 2024-05-15T12:55:18.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-15T12:55:18.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:55:19.294 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.1 2024-05-15T12:55:19.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-15T12:55:19.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:55:19.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:55:19.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:55:19.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:55:19.327 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:19.338+0000 7f5c528977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:55:19.358 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:19.370+0000 7f5c528977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:55:19.359 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:19.370+0000 7f5c528977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:55:19.743 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-15T12:55:19.744 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:55:19.744 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-15T12:55:19.744 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:55:19.744 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:55:19.744 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:19.744 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:55:19.744 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:55:19.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:19.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:55:19.935 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:19.946+0000 7f5c528977c0 -1 Falling back to public interface 2024-05-15T12:55:20.166 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:21.168 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:21.168 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:21.168 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:55:21.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:55:21.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:21.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:55:21.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:22.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:22.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:22.594 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:55:22.595 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:55:22.595 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:22.595 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:55:22.654 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:22.666+0000 7f5c528977c0 -1 osd.1 0 log_to_monitors true 2024-05-15T12:55:23.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:24.245 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:24.245 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:24.246 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:55:24.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:55:24.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:24.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:55:24.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:25.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:25.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:25.797 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:55:25.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:55:25.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:25.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:55:26.218 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:27.219 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:27.219 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:27.220 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:55:27.220 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:55:27.220 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:27.220 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:55:27.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:28.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:28.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:28.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:55:28.694 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:55:28.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:28.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:55:29.147 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:30.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:30.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:30.149 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:55:30.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:55:30.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:30.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:55:30.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:31.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:31.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:31.690 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:55:31.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:55:31.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:31.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:55:32.247 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:33.249 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:33.249 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:33.249 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:55:33.250 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:55:33.250 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:33.250 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:55:33.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:34.713 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:34.714 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:34.714 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:55:34.714 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:55:34.714 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:34.714 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:55:35.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:36.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:36.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:36.188 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:55:36.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:55:36.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:36.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:55:36.972 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:37.974 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:37.974 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:37.975 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:55:37.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:55:37.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:37.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:55:38.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:39.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:39.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:39.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:55:39.928 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:55:39.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:39.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:55:40.417 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:41.419 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:41.419 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:41.419 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:55:41.419 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:55:41.420 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:41.420 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:55:42.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:43.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:43.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:43.258 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:55:43.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:55:43.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:43.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:55:44.021 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:45.023 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:45.023 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:45.023 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:55:45.024 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:55:45.024 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:45.024 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:55:45.842 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:46.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:46.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:46.844 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:55:46.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:55:46.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:46.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:55:47.322 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:48.324 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:48.324 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:48.324 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T12:55:48.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T12:55:48.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:48.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T12:55:48.756 INFO:tasks.workunit.client.0.smithi005.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1485616801,v1:127.0.0.1:6811/1485616801] [v2:127.0.0.1:6812/1485616801,v1:127.0.0.1:6813/1485616801] exists,up 5278715e-854b-4e16-8706-edf249203c92 2024-05-15T12:55:48.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:55:48.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:55:48.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:55:48.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:55:48.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-15T12:55:48.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:55:48.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:55:48.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-15T12:55:48.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:55:48.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-15T12:55:48.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:55:48.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:55:48.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:55:48.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:55:48.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-15T12:55:48.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-15T12:55:48.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:55:48.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:55:48.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:55:48.761 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:55:48.761 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:55:48.761 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:55:48.761 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:55:48.761 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:55:48.762 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:55:48.762 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:55:48.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:55:48.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:55:48.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:55:48.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:55:48.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:55:48.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:55:48.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:55:48.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:55:48.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:55:48.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:55:48.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:55:48.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:55:48.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-15T12:55:48.763 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:55:48.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cc0c140b-1827-4ee4-bdb7-a7689f90b178 2024-05-15T12:55:48.763 INFO:tasks.workunit.client.0.smithi005.stdout:add osd2 cc0c140b-1827-4ee4-bdb7-a7689f90b178 2024-05-15T12:55:48.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 cc0c140b-1827-4ee4-bdb7-a7689f90b178' 2024-05-15T12:55:48.764 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:55:48.785 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDUsERmWuSULxAAjotQECBh7hHJQxhqdnTEXw== 2024-05-15T12:55:48.785 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDUsERmWuSULxAAjotQECBh7hHJQxhqdnTEXw=="}' 2024-05-15T12:55:48.785 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cc0c140b-1827-4ee4-bdb7-a7689f90b178 -i td/test-erasure-eio/2/new.json 2024-05-15T12:55:49.384 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:55:49.401 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-15T12:55:49.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDUsERmWuSULxAAjotQECBh7hHJQxhqdnTEXw== --osd-uuid cc0c140b-1827-4ee4-bdb7-a7689f90b178 2024-05-15T12:55:49.431 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:49.441+0000 7fde3b8cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:55:49.433 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:49.441+0000 7fde3b8cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:55:49.434 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:49.445+0000 7fde3b8cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:55:49.435 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:49.445+0000 7fde3b8cc7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T12:55:49.435 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:49.445+0000 7fde3b8cc7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T12:55:49.435 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:49.445+0000 7fde3b8cc7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T12:55:49.507 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:49.517+0000 7fde3b8cc7c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-15T12:55:55.894 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-15T12:55:55.894 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:55:55.895 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd2 key to auth repository 2024-05-15T12:55:55.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-15T12:55:55.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:55:56.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-15T12:55:56.462 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.2 2024-05-15T12:55:56.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:55:56.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:55:56.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:55:56.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:55:56.496 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:56.505+0000 7fdd294017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:55:56.506 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:56.517+0000 7fdd294017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:55:56.507 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:56.517+0000 7fdd294017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:55:56.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-15T12:55:56.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:55:56.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-15T12:55:56.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:55:56.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:55:56.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:56.873 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:55:56.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:55:56.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:56.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:55:57.047 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:57.057+0000 7fdd294017c0 -1 Falling back to public interface 2024-05-15T12:55:57.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:58.300 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:58.300 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:58.301 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:55:58.301 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:55:58.301 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:58.301 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:55:58.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:55:59.611 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:55:59.621+0000 7fdd294017c0 -1 osd.2 0 log_to_monitors true 2024-05-15T12:55:59.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:55:59.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:55:59.766 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:55:59.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:55:59.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:55:59.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:56:00.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:01.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:01.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:01.333 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:56:01.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:56:01.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:01.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:56:01.865 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:02.867 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:02.867 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:02.867 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:56:02.867 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:56:02.868 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:02.868 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:56:03.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:04.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:04.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:04.332 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:56:04.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:56:04.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:04.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:56:04.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:05.894 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:05.894 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:05.894 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:56:05.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:56:05.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:05.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:56:06.466 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:07.467 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:07.468 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:07.468 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:56:07.468 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:56:07.468 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:07.469 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:56:07.935 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:08.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:08.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:08.937 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:56:08.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:56:08.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:08.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:56:09.539 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:10.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:10.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:10.542 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:56:10.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:56:10.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:10.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:56:10.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:11.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:11.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:11.967 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:56:11.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:56:11.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:11.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:56:12.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:13.478 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:13.479 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:13.479 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:56:13.479 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:56:13.479 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:13.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:56:14.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:15.088 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:15.089 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:15.089 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:56:15.089 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:56:15.089 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:15.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:56:15.864 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:16.866 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:16.866 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:16.866 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:56:16.867 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:56:16.867 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:16.867 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:56:17.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:18.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:18.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:18.692 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:56:18.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:56:18.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:18.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:56:19.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:20.642 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:20.642 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:20.642 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:56:20.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:56:20.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:20.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:56:21.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:22.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:22.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:22.767 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:56:22.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:56:22.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:22.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:56:23.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:24.581 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:24.581 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:24.581 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:56:24.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:56:24.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:24.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T12:56:25.324 INFO:tasks.workunit.client.0.smithi005.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2439474863,v1:127.0.0.1:6819/2439474863] [v2:127.0.0.1:6820/2439474863,v1:127.0.0.1:6821/2439474863] exists,up cc0c140b-1827-4ee4-bdb7-a7689f90b178 2024-05-15T12:56:25.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:56:25.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:56:25.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:56:25.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:56:25.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-15T12:56:25.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:56:25.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:56:25.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-15T12:56:25.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:56:25.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-15T12:56:25.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:56:25.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:56:25.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:56:25.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:56:25.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-15T12:56:25.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-15T12:56:25.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:56:25.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:56:25.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:56:25.329 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:56:25.329 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:56:25.329 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:56:25.329 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:56:25.329 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:56:25.329 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:56:25.329 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:56:25.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:56:25.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:56:25.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:56:25.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:56:25.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:56:25.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:56:25.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:56:25.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:56:25.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:56:25.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:56:25.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:56:25.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:56:25.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-15T12:56:25.331 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:56:25.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1179f297-ae24-4950-8908-bdd186845ffc 2024-05-15T12:56:25.331 INFO:tasks.workunit.client.0.smithi005.stdout:add osd3 1179f297-ae24-4950-8908-bdd186845ffc 2024-05-15T12:56:25.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 1179f297-ae24-4950-8908-bdd186845ffc' 2024-05-15T12:56:25.332 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:56:25.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD5sERmmpeoFRAANobuT3PuzQaCRY6SkmAaog== 2024-05-15T12:56:25.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD5sERmmpeoFRAANobuT3PuzQaCRY6SkmAaog=="}' 2024-05-15T12:56:25.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1179f297-ae24-4950-8908-bdd186845ffc -i td/test-erasure-eio/3/new.json 2024-05-15T12:56:26.167 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:56:26.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-15T12:56:26.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD5sERmmpeoFRAANobuT3PuzQaCRY6SkmAaog== --osd-uuid 1179f297-ae24-4950-8908-bdd186845ffc 2024-05-15T12:56:26.215 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:56:26.225+0000 7f707a6277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:56:26.241 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:56:26.249+0000 7f707a6277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:56:26.242 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:56:26.253+0000 7f707a6277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:56:26.243 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:56:26.253+0000 7f707a6277c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T12:56:26.243 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:56:26.253+0000 7f707a6277c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T12:56:26.243 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:56:26.253+0000 7f707a6277c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T12:56:26.358 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:56:26.369+0000 7f707a6277c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-15T12:56:32.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-15T12:56:32.831 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:56:32.832 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd3 key to auth repository 2024-05-15T12:56:32.832 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-15T12:56:32.832 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:56:33.396 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.3 2024-05-15T12:56:33.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-15T12:56:33.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:56:33.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:56:33.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:56:33.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:56:33.422 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:56:33.432+0000 7f527f2707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:56:33.424 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:56:33.432+0000 7f527f2707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:56:33.425 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:56:33.432+0000 7f527f2707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:56:33.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-15T12:56:33.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:56:33.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-15T12:56:33.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:56:33.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:56:33.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:33.820 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:56:33.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:56:33.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:33.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:56:33.992 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:56:34.000+0000 7f527f2707c0 -1 Falling back to public interface 2024-05-15T12:56:34.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:35.238 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:35.238 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:35.238 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:56:35.239 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:56:35.239 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:35.239 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:56:35.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:36.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:36.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:36.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:56:36.670 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:56:36.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:36.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:56:36.672 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:56:36.680+0000 7f527f2707c0 -1 osd.3 0 log_to_monitors true 2024-05-15T12:56:37.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:38.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:38.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:38.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:56:38.188 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:56:38.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:38.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:56:38.897 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:39.898 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:39.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:39.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:56:39.899 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:56:39.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:39.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:56:40.468 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:41.470 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:41.470 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:41.471 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:56:41.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:56:41.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:41.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:56:41.947 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:42.948 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:42.949 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:42.949 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:56:42.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:56:42.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:42.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:56:43.524 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:44.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:44.526 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:56:44.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:44.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:56:44.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:44.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:56:44.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:45.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:45.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:45.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:56:45.962 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:56:45.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:45.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:56:46.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:47.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:47.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:47.427 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:56:47.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:56:47.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:47.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:56:47.885 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:48.887 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:48.887 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:48.887 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:56:48.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:56:48.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:48.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:56:49.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:50.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:50.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:50.310 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:56:50.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:56:50.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:50.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:56:51.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:52.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:52.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:52.289 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:56:52.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:56:52.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:52.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:56:53.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:54.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:54.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:54.267 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:56:54.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:56:54.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:54.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:56:54.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:55.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:55.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:55.814 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:56:55.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:56:55.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:55.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:56:56.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:57.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:57.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:57.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:56:57.355 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:56:57.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:57.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:56:58.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:56:59.197 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:56:59.197 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:56:59.197 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:56:59.198 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:56:59.198 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:56:59.198 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:57:00.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:01.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:01.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:01.155 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:57:01.156 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:57:01.156 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:01.156 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:57:02.122 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:03.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:03.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:03.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T12:57:03.125 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T12:57:03.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:03.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T12:57:03.552 INFO:tasks.workunit.client.0.smithi005.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3391334302,v1:127.0.0.1:6827/3391334302] [v2:127.0.0.1:6828/3391334302,v1:127.0.0.1:6829/3391334302] exists,up 1179f297-ae24-4950-8908-bdd186845ffc 2024-05-15T12:57:03.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:57:03.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:57:03.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:57:03.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:57:03.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-05-15T12:57:03.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:57:03.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:57:03.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-05-15T12:57:03.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:57:03.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-05-15T12:57:03.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:57:03.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:57:03.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:57:03.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:57:03.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-05-15T12:57:03.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-05-15T12:57:03.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:57:03.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:57:03.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:57:03.555 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:57:03.555 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:57:03.555 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:57:03.555 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:57:03.555 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:57:03.555 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:57:03.556 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:57:03.556 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:57:03.556 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:57:03.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:57:03.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:57:03.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:57:03.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:57:03.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:57:03.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:57:03.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:57:03.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:57:03.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:57:03.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:57:03.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-05-15T12:57:03.557 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:57:03.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=17ea1375-30b5-4678-881e-d30d55d870df 2024-05-15T12:57:03.558 INFO:tasks.workunit.client.0.smithi005.stdout:add osd4 17ea1375-30b5-4678-881e-d30d55d870df 2024-05-15T12:57:03.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 17ea1375-30b5-4678-881e-d30d55d870df' 2024-05-15T12:57:03.559 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:57:03.578 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAfsURmGGNCIxAA21PPJ3oLffnnbTT9sb+c8A== 2024-05-15T12:57:03.578 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAfsURmGGNCIxAA21PPJ3oLffnnbTT9sb+c8A=="}' 2024-05-15T12:57:03.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 17ea1375-30b5-4678-881e-d30d55d870df -i td/test-erasure-eio/4/new.json 2024-05-15T12:57:04.183 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:57:04.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-05-15T12:57:04.202 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAfsURmGGNCIxAA21PPJ3oLffnnbTT9sb+c8A== --osd-uuid 17ea1375-30b5-4678-881e-d30d55d870df 2024-05-15T12:57:04.239 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:04.252+0000 7fc89a61b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:57:04.241 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:04.256+0000 7fc89a61b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:57:04.243 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:04.256+0000 7fc89a61b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:57:04.243 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:04.256+0000 7fc89a61b7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T12:57:04.244 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:04.256+0000 7fc89a61b7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T12:57:04.244 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:04.256+0000 7fc89a61b7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T12:57:04.354 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:04.364+0000 7fc89a61b7c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-05-15T12:57:12.294 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-05-15T12:57:12.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:57:12.296 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd4 key to auth repository 2024-05-15T12:57:12.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-05-15T12:57:12.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:57:12.894 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.4 2024-05-15T12:57:12.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-05-15T12:57:12.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:57:12.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:57:12.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:57:12.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:57:12.929 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:12.939+0000 7fc5389487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:57:12.932 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:12.943+0000 7fc5389487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:57:12.933 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:12.943+0000 7fc5389487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:57:13.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-05-15T12:57:13.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:57:13.321 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-05-15T12:57:13.321 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:57:13.321 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:57:13.321 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:13.321 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:57:13.321 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:57:13.322 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:13.322 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:57:13.480 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:13.491+0000 7fc5389487c0 -1 Falling back to public interface 2024-05-15T12:57:13.904 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:14.906 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:14.906 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:14.906 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:57:14.906 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:57:14.907 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:14.907 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:57:15.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:16.174 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:16.183+0000 7fc5389487c0 -1 osd.4 0 log_to_monitors true 2024-05-15T12:57:16.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:16.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:16.337 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:57:16.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:57:16.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:16.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:57:16.997 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:17.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:17.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:17.999 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:57:18.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:57:18.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:18.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:57:18.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:19.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:19.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:19.763 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:57:19.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:57:19.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:19.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:57:20.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:21.442 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:21.442 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:21.442 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:57:21.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:57:21.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:21.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:57:21.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:22.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:22.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:22.929 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:57:22.929 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:57:22.929 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:22.929 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:57:23.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:24.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:24.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:24.505 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:57:24.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:57:24.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:24.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:57:25.198 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:26.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:26.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:26.201 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:57:26.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:57:26.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:26.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:57:26.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:27.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:27.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:27.638 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:57:27.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:57:27.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:27.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:57:28.210 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:29.212 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:29.212 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:29.212 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:57:29.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:57:29.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:29.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:57:29.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:30.642 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:30.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:30.643 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:57:30.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:57:30.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:30.644 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:57:31.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:32.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:32.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:32.076 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:57:32.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:57:32.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:32.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:57:33.167 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:34.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:34.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:34.169 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:57:34.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:57:34.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:34.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:57:35.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:36.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:36.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:36.061 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:57:36.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:57:36.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:36.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:57:37.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:38.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:38.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:38.447 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:57:38.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:57:38.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:38.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:57:39.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:40.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:40.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:40.289 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:57:40.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:57:40.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:40.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:57:41.078 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:42.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:42.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:42.080 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:57:42.080 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:57:42.080 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:42.080 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T12:57:42.609 INFO:tasks.workunit.client.0.smithi005.stdout:osd.4 up in weight 1 up_from 41 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3379825185,v1:127.0.0.1:6835/3379825185] [v2:127.0.0.1:6836/3379825185,v1:127.0.0.1:6837/3379825185] exists,up 17ea1375-30b5-4678-881e-d30d55d870df 2024-05-15T12:57:42.610 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:57:42.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:57:42.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:57:42.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:57:42.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-05-15T12:57:42.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:57:42.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:57:42.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-05-15T12:57:42.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:57:42.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-05-15T12:57:42.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:57:42.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:57:42.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:57:42.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:57:42.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-05-15T12:57:42.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-05-15T12:57:42.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:57:42.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:57:42.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:57:42.612 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:57:42.612 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:57:42.612 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:57:42.612 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:57:42.612 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:57:42.612 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:57:42.612 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:57:42.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:57:42.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:57:42.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:57:42.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:57:42.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:57:42.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:57:42.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:57:42.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:57:42.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:57:42.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:57:42.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:57:42.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:57:42.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-05-15T12:57:42.614 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:57:42.615 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=05043446-bc04-474a-8bd5-c835d256e072 2024-05-15T12:57:42.615 INFO:tasks.workunit.client.0.smithi005.stdout:add osd5 05043446-bc04-474a-8bd5-c835d256e072 2024-05-15T12:57:42.615 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 05043446-bc04-474a-8bd5-c835d256e072' 2024-05-15T12:57:42.615 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:57:42.637 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGsURmmrnIJhAAmamVDVLh987Cdaw49sNxKQ== 2024-05-15T12:57:42.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGsURmmrnIJhAAmamVDVLh987Cdaw49sNxKQ=="}' 2024-05-15T12:57:42.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 05043446-bc04-474a-8bd5-c835d256e072 -i td/test-erasure-eio/5/new.json 2024-05-15T12:57:43.592 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:57:43.615 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-05-15T12:57:43.616 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGsURmmrnIJhAAmamVDVLh987Cdaw49sNxKQ== --osd-uuid 05043446-bc04-474a-8bd5-c835d256e072 2024-05-15T12:57:43.647 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:43.659+0000 7f5bad83c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:57:43.732 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:43.743+0000 7f5bad83c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:57:43.734 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:43.747+0000 7f5bad83c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:57:43.735 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:43.747+0000 7f5bad83c7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-15T12:57:43.735 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:43.747+0000 7f5bad83c7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-15T12:57:43.735 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:43.747+0000 7f5bad83c7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-15T12:57:43.873 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:43.883+0000 7f5bad83c7c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-05-15T12:57:51.164 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-05-15T12:57:51.164 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:57:51.165 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd5 key to auth repository 2024-05-15T12:57:51.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-05-15T12:57:51.166 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:57:51.724 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.5 2024-05-15T12:57:51.725 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-05-15T12:57:51.725 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:57:51.725 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:57:51.725 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:57:51.725 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:57:51.757 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:51.767+0000 7f0c044a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:57:51.771 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:51.783+0000 7f0c044a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:57:51.772 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:51.783+0000 7f0c044a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:57:52.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-05-15T12:57:52.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:57:52.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-05-15T12:57:52.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:57:52.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:57:52.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:52.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:57:52.195 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:57:52.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:52.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:57:52.335 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:52.346+0000 7f0c044a97c0 -1 Falling back to public interface 2024-05-15T12:57:52.615 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:53.617 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:53.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:53.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:57:53.618 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:57:53.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:53.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:57:54.029 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:54.911 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:57:54.922+0000 7f0c044a97c0 -1 osd.5 0 log_to_monitors true 2024-05-15T12:57:55.031 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:55.031 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:55.031 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:57:55.032 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:57:55.032 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:55.032 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:57:55.621 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:56.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:56.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:56.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:57:56.623 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:57:56.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:56.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:57:57.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:58.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:58.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:58.308 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:57:58.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:57:58.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:58.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:57:58.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:57:59.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:57:59.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:57:59.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:57:59.735 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:57:59.736 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:57:59.736 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:58:00.255 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:01.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:01.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:01.257 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:58:01.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:58:01.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:01.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:58:01.776 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:02.777 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:02.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:02.778 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:58:02.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:58:02.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:02.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:58:03.223 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:04.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:04.225 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:04.225 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:58:04.225 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:58:04.225 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:04.225 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:58:04.832 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:05.833 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:05.833 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:05.834 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:58:05.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:58:05.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:05.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:58:06.284 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:07.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:07.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:07.287 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:58:07.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:58:07.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:07.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:58:07.773 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:08.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:08.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:08.775 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:58:08.776 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:58:08.776 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:08.776 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:58:09.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:10.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:10.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:10.218 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:58:10.218 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:58:10.218 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:10.218 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:58:11.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:12.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:12.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:12.011 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:58:12.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:58:12.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:12.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:58:13.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:14.078 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:14.078 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:14.078 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:58:14.079 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:58:14.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:14.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:58:14.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:15.741 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:15.741 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:15.742 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:58:15.742 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:58:15.742 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:15.742 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:58:16.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:17.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:17.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:17.587 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:58:17.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:58:17.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:17.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:58:18.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:19.162 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:19.162 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:19.162 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:58:19.163 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:58:19.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:19.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:58:20.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:21.191 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:21.191 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:21.191 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T12:58:21.192 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T12:58:21.192 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:21.192 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T12:58:21.624 INFO:tasks.workunit.client.0.smithi005.stdout:osd.5 up in weight 1 up_from 51 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1390252781,v1:127.0.0.1:6843/1390252781] [v2:127.0.0.1:6844/1390252781,v1:127.0.0.1:6845/1390252781] exists,up 05043446-bc04-474a-8bd5-c835d256e072 2024-05-15T12:58:21.626 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:58:21.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:58:21.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:58:21.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:58:21.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-05-15T12:58:21.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:58:21.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:58:21.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-05-15T12:58:21.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:58:21.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-05-15T12:58:21.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:58:21.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:58:21.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:58:21.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:58:21.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-05-15T12:58:21.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-05-15T12:58:21.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:58:21.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:58:21.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:58:21.628 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:58:21.628 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:58:21.628 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:58:21.630 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:58:21.630 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:58:21.630 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:58:21.630 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:58:21.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:58:21.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:58:21.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:58:21.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:58:21.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:58:21.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:58:21.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:58:21.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:58:21.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:58:21.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:58:21.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:58:21.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:58:21.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-05-15T12:58:21.631 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:58:21.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=537786b0-0731-4df9-a701-25242b5df88b 2024-05-15T12:58:21.632 INFO:tasks.workunit.client.0.smithi005.stdout:add osd6 537786b0-0731-4df9-a701-25242b5df88b 2024-05-15T12:58:21.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 537786b0-0731-4df9-a701-25242b5df88b' 2024-05-15T12:58:21.632 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:58:21.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBtsURmkHrGJxAAax8nozOI32Fx+3Pg+fo5LA== 2024-05-15T12:58:21.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBtsURmkHrGJxAAax8nozOI32Fx+3Pg+fo5LA=="}' 2024-05-15T12:58:21.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 537786b0-0731-4df9-a701-25242b5df88b -i td/test-erasure-eio/6/new.json 2024-05-15T12:58:22.169 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:58:22.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-05-15T12:58:22.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBtsURmkHrGJxAAax8nozOI32Fx+3Pg+fo5LA== --osd-uuid 537786b0-0731-4df9-a701-25242b5df88b 2024-05-15T12:58:22.225 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:58:22.234+0000 7f1a2eb127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:58:22.262 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:58:22.274+0000 7f1a2eb127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:58:22.263 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:58:22.274+0000 7f1a2eb127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:58:22.264 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:58:22.274+0000 7f1a2eb127c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-15T12:58:22.264 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:58:22.274+0000 7f1a2eb127c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-15T12:58:22.264 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:58:22.274+0000 7f1a2eb127c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-15T12:58:22.395 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:58:22.406+0000 7f1a2eb127c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-05-15T12:58:30.143 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-05-15T12:58:30.143 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:58:30.146 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd6 key to auth repository 2024-05-15T12:58:30.146 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-05-15T12:58:30.146 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:58:30.711 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.6 2024-05-15T12:58:30.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-05-15T12:58:30.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:58:30.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:58:30.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:58:30.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:58:30.744 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:58:30.754+0000 7f1dafb8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:58:30.780 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:58:30.790+0000 7f1dafb8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:58:30.781 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:58:30.790+0000 7f1dafb8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:58:31.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-05-15T12:58:31.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:58:31.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-05-15T12:58:31.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:58:31.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:58:31.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:31.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:58:31.179 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:58:31.180 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:58:31.180 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:31.339 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:58:31.350+0000 7f1dafb8f7c0 -1 Falling back to public interface 2024-05-15T12:58:31.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:32.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:32.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:32.614 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:58:32.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:58:32.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:32.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:58:33.088 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:33.944 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:58:33.954+0000 7f1dafb8f7c0 -1 osd.6 0 log_to_monitors true 2024-05-15T12:58:34.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:34.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:34.091 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:58:34.091 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:58:34.091 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:34.091 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:58:34.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:35.596 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:35.596 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:35.596 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:58:35.597 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:58:35.597 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:35.597 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:58:36.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:37.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:37.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:37.237 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:58:37.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:58:37.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:37.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:58:37.702 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:38.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:38.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:38.705 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:58:38.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:58:38.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:38.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:58:39.177 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:40.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:40.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:40.179 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:58:40.180 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:58:40.180 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:40.180 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:58:40.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:41.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:41.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:41.630 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:58:41.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:58:41.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:41.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:58:42.139 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:43.141 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:43.141 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:43.142 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:58:43.142 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:58:43.142 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:43.142 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:58:43.576 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:44.578 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:44.578 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:44.578 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:58:44.578 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:58:44.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:44.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:58:45.038 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:46.039 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:46.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:46.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:58:46.040 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:58:46.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:46.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:58:46.598 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:47.598 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:47.598 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:47.598 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:58:47.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:58:47.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:47.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:58:48.051 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:49.053 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:49.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:49.054 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:58:49.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:58:49.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:49.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:58:49.629 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:50.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:50.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:50.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:58:50.631 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:58:50.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:50.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:58:51.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:52.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:52.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:52.483 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:58:52.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:58:52.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:52.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:58:53.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:54.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:54.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:54.604 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:58:54.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:58:54.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:54.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:58:55.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:56.410 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:56.410 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:56.411 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:58:56.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:58:56.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:56.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:58:57.410 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:58:58.412 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:58:58.413 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:58:58.413 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:58:58.413 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:58:58.413 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:58:58.413 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:58:59.051 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:00.053 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:00.053 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:00.053 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T12:59:00.053 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T12:59:00.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:00.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T12:59:00.560 INFO:tasks.workunit.client.0.smithi005.stdout:osd.6 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/287679114,v1:127.0.0.1:6851/287679114] [v2:127.0.0.1:6852/287679114,v1:127.0.0.1:6853/287679114] exists,up 537786b0-0731-4df9-a701-25242b5df88b 2024-05-15T12:59:00.563 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:59:00.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:59:00.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:59:00.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:59:00.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 7 2024-05-15T12:59:00.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:59:00.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:59:00.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-05-15T12:59:00.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:59:00.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/7 2024-05-15T12:59:00.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:59:00.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:59:00.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:59:00.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:59:00.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/7' 2024-05-15T12:59:00.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/7/journal' 2024-05-15T12:59:00.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:59:00.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:59:00.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:59:00.565 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:59:00.565 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:59:00.565 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:59:00.565 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:59:00.565 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:59:00.565 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:59:00.565 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:59:00.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:59:00.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:59:00.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:59:00.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:59:00.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:59:00.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:59:00.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:59:00.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:59:00.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:59:00.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:59:00.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:59:00.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:59:00.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/7 2024-05-15T12:59:00.567 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:59:00.567 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2c128fc2-7a37-4629-8219-f178d582c845 2024-05-15T12:59:00.567 INFO:tasks.workunit.client.0.smithi005.stdout:add osd7 2c128fc2-7a37-4629-8219-f178d582c845 2024-05-15T12:59:00.567 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 2c128fc2-7a37-4629-8219-f178d582c845' 2024-05-15T12:59:00.568 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:59:00.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCUsURmoqigIxAAViBwP2GpeEEdYcLoTR9ZRg== 2024-05-15T12:59:00.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCUsURmoqigIxAAViBwP2GpeEEdYcLoTR9ZRg=="}' 2024-05-15T12:59:00.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2c128fc2-7a37-4629-8219-f178d582c845 -i td/test-erasure-eio/7/new.json 2024-05-15T12:59:01.453 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:59:01.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/7/new.json 2024-05-15T12:59:01.474 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCUsURmoqigIxAAViBwP2GpeEEdYcLoTR9ZRg== --osd-uuid 2c128fc2-7a37-4629-8219-f178d582c845 2024-05-15T12:59:01.511 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:01.521+0000 7f3efa3547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:59:01.593 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:01.601+0000 7f3efa3547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:59:01.595 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:01.605+0000 7f3efa3547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:59:01.596 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:01.605+0000 7f3efa3547c0 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-05-15T12:59:01.596 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:01.605+0000 7f3efa3547c0 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-05-15T12:59:01.596 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:01.605+0000 7f3efa3547c0 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-05-15T12:59:01.735 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:01.745+0000 7f3efa3547c0 -1 bluestore(td/test-erasure-eio/7) _read_fsid unparsable uuid 2024-05-15T12:59:08.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/7/keyring 2024-05-15T12:59:08.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:59:08.951 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd7 key to auth repository 2024-05-15T12:59:08.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-05-15T12:59:08.952 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:59:09.512 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.7 2024-05-15T12:59:09.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-05-15T12:59:09.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:59:09.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:59:09.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:59:09.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:59:09.539 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:09.549+0000 7f36ecf867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:59:09.557 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:09.565+0000 7f36ecf867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:59:09.558 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:09.569+0000 7f36ecf867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:59:09.929 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-05-15T12:59:09.929 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:59:09.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-05-15T12:59:09.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:59:09.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:59:09.930 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:59:09.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:09.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:59:09.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:09.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:59:10.120 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:10.129+0000 7f36ecf867c0 -1 Falling back to public interface 2024-05-15T12:59:10.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:11.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:11.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:11.365 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:59:11.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:59:11.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:11.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:59:11.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:12.799 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:12.799 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:12.800 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:59:12.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:59:12.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:12.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:59:12.826 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:12.837+0000 7f36ecf867c0 -1 osd.7 0 log_to_monitors true 2024-05-15T12:59:13.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:14.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:14.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:14.561 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:59:14.561 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:59:14.561 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:14.561 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:59:15.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:16.277 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:16.277 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:16.277 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:59:16.278 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:59:16.278 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:16.278 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:59:16.742 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:17.744 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:17.744 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:17.745 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:59:17.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:59:17.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:17.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:59:18.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:19.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:19.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:19.282 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:59:19.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:59:19.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:19.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:59:19.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:20.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:20.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:20.712 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T12:59:20.713 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T12:59:20.713 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:20.713 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:59:21.274 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:22.277 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:22.277 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:22.277 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T12:59:22.277 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:59:22.277 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:22.278 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:59:22.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:23.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:23.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:23.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T12:59:23.712 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T12:59:23.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:23.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:59:24.247 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:25.249 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:25.249 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:25.249 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T12:59:25.250 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T12:59:25.250 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:25.250 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:59:25.674 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:26.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:26.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:26.676 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T12:59:26.677 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T12:59:26.677 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:26.677 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:59:27.220 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:28.221 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:28.222 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:28.222 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T12:59:28.222 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T12:59:28.222 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:28.222 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:59:29.302 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:30.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:30.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:30.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T12:59:30.304 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T12:59:30.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:30.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:59:30.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:31.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:31.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:31.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T12:59:31.796 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T12:59:31.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:31.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:59:32.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:33.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:33.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:33.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T12:59:33.394 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T12:59:33.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:33.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:59:33.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:34.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:34.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:34.845 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T12:59:34.845 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T12:59:34.845 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:34.845 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:59:35.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:36.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:36.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:36.540 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T12:59:36.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T12:59:36.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:36.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:59:37.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:38.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:38.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:38.055 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T12:59:38.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T12:59:38.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:38.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:59:38.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:39.551 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:39.551 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:39.551 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-15T12:59:39.551 INFO:tasks.workunit.client.0.smithi005.stdout:19 2024-05-15T12:59:39.552 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:39.552 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-15T12:59:40.069 INFO:tasks.workunit.client.0.smithi005.stdout:osd.7 up in weight 1 up_from 70 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/1767854578,v1:127.0.0.1:6859/1767854578] [v2:127.0.0.1:6860/1767854578,v1:127.0.0.1:6861/1767854578] exists,up 2c128fc2-7a37-4629-8219-f178d582c845 2024-05-15T12:59:40.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T12:59:40.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T12:59:40.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T12:59:40.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T12:59:40.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 8 2024-05-15T12:59:40.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T12:59:40.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T12:59:40.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-05-15T12:59:40.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T12:59:40.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/8 2024-05-15T12:59:40.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T12:59:40.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T12:59:40.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T12:59:40.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T12:59:40.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/8' 2024-05-15T12:59:40.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/8/journal' 2024-05-15T12:59:40.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T12:59:40.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T12:59:40.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T12:59:40.075 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T12:59:40.075 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T12:59:40.075 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T12:59:40.075 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T12:59:40.075 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T12:59:40.075 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T12:59:40.075 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:59:40.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T12:59:40.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T12:59:40.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T12:59:40.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T12:59:40.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T12:59:40.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T12:59:40.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T12:59:40.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T12:59:40.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T12:59:40.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T12:59:40.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T12:59:40.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T12:59:40.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/8 2024-05-15T12:59:40.077 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T12:59:40.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0aacf6a0-f6b4-4564-98f4-51c142a946ff 2024-05-15T12:59:40.077 INFO:tasks.workunit.client.0.smithi005.stdout:add osd8 0aacf6a0-f6b4-4564-98f4-51c142a946ff 2024-05-15T12:59:40.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 0aacf6a0-f6b4-4564-98f4-51c142a946ff' 2024-05-15T12:59:40.077 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T12:59:40.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC8sURmo8KbBhAArm2daJp4W8yiQRmF+Nrh6w== 2024-05-15T12:59:40.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC8sURmo8KbBhAArm2daJp4W8yiQRmF+Nrh6w=="}' 2024-05-15T12:59:40.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0aacf6a0-f6b4-4564-98f4-51c142a946ff -i td/test-erasure-eio/8/new.json 2024-05-15T12:59:41.335 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T12:59:41.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/8/new.json 2024-05-15T12:59:41.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC8sURmo8KbBhAArm2daJp4W8yiQRmF+Nrh6w== --osd-uuid 0aacf6a0-f6b4-4564-98f4-51c142a946ff 2024-05-15T12:59:41.393 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:41.400+0000 7f87b73407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:59:41.399 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:41.408+0000 7f87b73407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:59:41.400 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:41.408+0000 7f87b73407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:59:41.401 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:41.408+0000 7f87b73407c0 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-05-15T12:59:41.401 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:41.408+0000 7f87b73407c0 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-05-15T12:59:41.401 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:41.408+0000 7f87b73407c0 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-05-15T12:59:41.566 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:41.576+0000 7f87b73407c0 -1 bluestore(td/test-erasure-eio/8) _read_fsid unparsable uuid 2024-05-15T12:59:49.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/8/keyring 2024-05-15T12:59:49.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T12:59:49.608 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd8 key to auth repository 2024-05-15T12:59:49.609 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-05-15T12:59:49.609 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T12:59:50.169 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.8 2024-05-15T12:59:50.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-05-15T12:59:50.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T12:59:50.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T12:59:50.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T12:59:50.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T12:59:50.202 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:50.212+0000 7f9e96e8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:59:50.205 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:50.212+0000 7f9e96e8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:59:50.206 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:50.216+0000 7f9e96e8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T12:59:50.617 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-05-15T12:59:50.617 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T12:59:50.617 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-05-15T12:59:50.617 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T12:59:50.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T12:59:50.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:50.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T12:59:50.618 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T12:59:50.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:50.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:59:50.755 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:50.764+0000 7f9e96e8a7c0 -1 Falling back to public interface 2024-05-15T12:59:51.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:52.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:52.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:52.079 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T12:59:52.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T12:59:52.080 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:52.080 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:59:52.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:53.461 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T12:59:53.468+0000 7f9e96e8a7c0 -1 osd.8 0 log_to_monitors true 2024-05-15T12:59:53.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:53.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:53.566 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T12:59:53.567 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T12:59:53.567 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:53.567 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:59:54.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:55.127 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:55.127 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:55.128 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T12:59:55.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T12:59:55.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:55.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:59:55.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:56.831 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:56.832 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:56.832 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T12:59:56.833 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T12:59:56.833 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:56.833 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:59:57.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:58.398 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:58.398 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:58.398 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T12:59:58.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T12:59:58.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:58.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T12:59:58.847 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T12:59:59.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T12:59:59.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T12:59:59.849 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T12:59:59.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T12:59:59.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T12:59:59.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T13:00:00.278 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:00:01.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:00:01.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:00:01.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:00:01.280 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:00:01.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:00:01.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T13:00:01.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:00:02.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:00:02.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:00:02.758 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:00:02.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:00:02.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:00:02.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T13:00:03.177 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:00:04.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:00:04.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:00:04.180 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:00:04.180 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:00:04.180 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:00:04.180 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T13:00:04.621 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:00:05.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:00:05.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:00:05.623 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:00:05.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:00:05.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:00:05.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T13:00:06.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:00:07.118 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:00:07.118 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:00:07.119 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:00:07.119 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:00:07.119 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:00:07.119 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T13:00:07.576 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:00:08.578 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:00:08.578 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:00:08.578 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:00:08.578 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:00:08.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:00:08.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T13:00:09.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:00:10.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:00:10.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:00:10.172 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:00:10.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:00:10.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:00:10.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T13:00:11.467 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:00:12.469 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:00:12.469 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:00:12.469 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:00:12.469 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:00:12.470 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:00:12.470 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T13:00:13.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:00:14.322 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:00:14.322 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:00:14.322 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:00:14.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:00:14.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:00:14.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T13:00:15.249 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:00:16.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:00:16.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:00:16.251 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:00:16.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:00:16.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:00:16.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T13:00:17.141 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:00:18.142 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:00:18.142 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:00:18.143 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:00:18.143 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:00:18.143 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:00:18.143 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T13:00:18.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:00:19.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:00:19.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:00:19.625 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:00:19.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:00:19.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:00:19.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T13:00:20.209 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:00:21.208 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:00:21.208 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:00:21.209 INFO:tasks.workunit.client.0.smithi005.stdout:19 2024-05-15T13:00:21.209 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-15T13:00:21.209 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:00:21.209 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-15T13:00:21.983 INFO:tasks.workunit.client.0.smithi005.stdout:osd.8 up in weight 1 up_from 81 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/973557424,v1:127.0.0.1:6867/973557424] [v2:127.0.0.1:6868/973557424,v1:127.0.0.1:6869/973557424] exists,up 0aacf6a0-f6b4-4564-98f4-51c142a946ff 2024-05-15T13:00:21.984 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:00:21.984 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:00:21.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:00:21.986 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-15T13:00:21.987 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T13:00:21.987 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T13:00:21.987 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:00:21.987 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:00:21.987 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:00:21.987 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.0.asok 2024-05-15T13:00:21.987 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-15T13:00:21.987 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-osd.0.asok log flush 2024-05-15T13:00:22.104 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T13:00:22.118 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-15T13:00:22.128 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T12:54:44.098+0000 7f5df3e5f7c0 0 load: jerasure load: lrc 2024-05-15T13:00:22.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:423: TEST_ec_recovery_multiple_errors: local poolname=pool-jerasure 2024-05-15T13:00:22.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:424: TEST_ec_recovery_multiple_errors: create_erasure_coded_pool pool-jerasure 4 4 2024-05-15T13:00:22.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-15T13:00:22.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-15T13:00:22.130 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=4 2024-05-15T13:00:22.130 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-15T13:00:22.130 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=4 2024-05-15T13:00:22.130 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-15T13:00:22.130 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=4 m=4 crush-failure-domain=osd 2024-05-15T13:00:23.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-15T13:00:23.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-15T13:00:24.998 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-jerasure' already exists 2024-05-15T13:00:25.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T13:00:26.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-15T13:00:26.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:00:26.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:00:26.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:00:26.021 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:00:26.022 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:00:26.022 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:00:26.022 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:00:26.022 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:00:26.022 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:00:26.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:00:26.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:00:26.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:00:26.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:00:26.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:00:26.150 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:00:26.713 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:00:26.714 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:00:26.714 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:00:26.714 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T13:00:26.714 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T13:00:26.714 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T13:00:26.714 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T13:00:26.714 INFO:tasks.workunit.client.0.smithi005.stderr:7 2024-05-15T13:00:26.714 INFO:tasks.workunit.client.0.smithi005.stderr:8' 2024-05-15T13:00:26.714 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:00:26.714 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:00:26.714 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:00:26.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803847 2024-05-15T13:00:26.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803847 2024-05-15T13:00:26.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803847' 2024-05-15T13:00:26.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:00:26.884 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:00:27.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574912 2024-05-15T13:00:27.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574912 2024-05-15T13:00:27.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803847 1-55834574912' 2024-05-15T13:00:27.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:00:27.043 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:00:27.198 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378681 2024-05-15T13:00:27.198 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378681 2024-05-15T13:00:27.198 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803847 1-55834574912 2-81604378681' 2024-05-15T13:00:27.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:00:27.199 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:00:27.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084336 2024-05-15T13:00:27.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084336 2024-05-15T13:00:27.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803847 1-55834574912 2-81604378681 3-120259084336' 2024-05-15T13:00:27.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:00:27.429 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T13:00:27.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659172 2024-05-15T13:00:27.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659172 2024-05-15T13:00:27.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803847 1-55834574912 2-81604378681 3-120259084336 4-176093659172' 2024-05-15T13:00:27.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:00:27.592 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-15T13:00:27.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=219043332128 2024-05-15T13:00:27.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 219043332128 2024-05-15T13:00:27.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803847 1-55834574912 2-81604378681 3-120259084336 4-176093659172 5-219043332128' 2024-05-15T13:00:27.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:00:27.757 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-15T13:00:27.909 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005081 2024-05-15T13:00:27.909 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005081 2024-05-15T13:00:27.909 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803847 1-55834574912 2-81604378681 3-120259084336 4-176093659172 5-219043332128 6-261993005081' 2024-05-15T13:00:27.909 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:00:27.909 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-15T13:00:28.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710736 2024-05-15T13:00:28.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710736 2024-05-15T13:00:28.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803847 1-55834574912 2-81604378681 3-120259084336 4-176093659172 5-219043332128 6-261993005081 7-300647710736' 2024-05-15T13:00:28.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:00:28.056 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-15T13:00:28.212 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350983 2024-05-15T13:00:28.212 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350983 2024-05-15T13:00:28.212 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803847 1-55834574912 2-81604378681 3-120259084336 4-176093659172 5-219043332128 6-261993005081 7-300647710736 8-347892350983' 2024-05-15T13:00:28.212 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:00:28.213 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:00:28.213 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803847 2024-05-15T13:00:28.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:00:28.214 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803847 2024-05-15T13:00:28.214 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:00:28.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803847 2024-05-15T13:00:28.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803847' 2024-05-15T13:00:28.215 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 25769803847 2024-05-15T13:00:28.216 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:00:28.620 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803847 -lt 25769803847 2024-05-15T13:00:28.620 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:00:28.621 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574912 2024-05-15T13:00:28.621 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:00:28.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:00:28.623 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574912 2024-05-15T13:00:28.623 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:00:28.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574912 2024-05-15T13:00:28.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574912' 2024-05-15T13:00:28.624 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 55834574912 2024-05-15T13:00:28.624 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:00:29.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574912 -lt 55834574912 2024-05-15T13:00:29.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:00:29.051 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378681 2024-05-15T13:00:29.051 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:00:29.053 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:00:29.054 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:00:29.054 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378681 2024-05-15T13:00:29.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378681 2024-05-15T13:00:29.055 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 81604378681 2024-05-15T13:00:29.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378681' 2024-05-15T13:00:29.056 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:00:29.466 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378681 -lt 81604378681 2024-05-15T13:00:29.466 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:00:29.467 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084336 2024-05-15T13:00:29.467 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:00:29.469 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:00:29.469 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084336 2024-05-15T13:00:29.470 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:00:29.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084336 2024-05-15T13:00:29.471 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 120259084336 2024-05-15T13:00:29.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084336' 2024-05-15T13:00:29.471 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:00:29.910 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084336 -lt 120259084336 2024-05-15T13:00:29.910 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:00:29.910 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-176093659172 2024-05-15T13:00:29.910 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:00:29.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T13:00:29.913 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-176093659172 2024-05-15T13:00:29.913 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:00:29.914 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659172 2024-05-15T13:00:29.915 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.4 seq 176093659172 2024-05-15T13:00:29.915 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 176093659172' 2024-05-15T13:00:29.915 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T13:00:30.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659173 -lt 176093659172 2024-05-15T13:00:30.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:00:30.359 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-219043332128 2024-05-15T13:00:30.359 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:00:30.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-15T13:00:30.362 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:00:30.362 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-219043332128 2024-05-15T13:00:30.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=219043332128 2024-05-15T13:00:30.363 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.5 seq 219043332128 2024-05-15T13:00:30.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 219043332128' 2024-05-15T13:00:30.363 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-15T13:00:30.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 219043332128 -lt 219043332128 2024-05-15T13:00:30.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:00:30.809 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-261993005081 2024-05-15T13:00:30.810 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:00:30.811 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-15T13:00:30.811 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-261993005081 2024-05-15T13:00:30.812 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:00:30.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005081 2024-05-15T13:00:30.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 261993005081' 2024-05-15T13:00:30.813 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.6 seq 261993005081 2024-05-15T13:00:30.814 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-15T13:00:31.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005081 -lt 261993005081 2024-05-15T13:00:31.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:00:31.254 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-300647710736 2024-05-15T13:00:31.255 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:00:31.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-05-15T13:00:31.257 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-300647710736 2024-05-15T13:00:31.257 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:00:31.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710736 2024-05-15T13:00:31.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 300647710736' 2024-05-15T13:00:31.259 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.7 seq 300647710736 2024-05-15T13:00:31.259 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-05-15T13:00:31.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710737 -lt 300647710736 2024-05-15T13:00:31.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:00:31.695 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-347892350983 2024-05-15T13:00:31.695 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:00:31.697 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-15T13:00:31.697 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-347892350983 2024-05-15T13:00:31.698 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:00:31.699 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350983 2024-05-15T13:00:31.699 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.8 seq 347892350983 2024-05-15T13:00:31.699 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 347892350983' 2024-05-15T13:00:31.699 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-15T13:00:32.122 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350983 -lt 347892350983 2024-05-15T13:00:32.122 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:00:32.123 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:00:32.123 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:00:32.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:00:32.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:00:32.667 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:00:32.667 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:00:32.667 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:00:32.667 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:00:32.667 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:00:32.668 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:00:33.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:00:33.117 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:00:33.117 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:00:33.117 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:00:33.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:00:33.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:00:33.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:00:33.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:426: TEST_ec_recovery_multiple_errors: rados_put td/test-erasure-eio pool-jerasure myobject 2024-05-15T13:00:33.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-15T13:00:33.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-15T13:00:33.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-05-15T13:00:33.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:00:33.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-15T13:00:33.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:00:33.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-15T13:00:33.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:00:33.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-15T13:00:33.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:00:33.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-15T13:00:33.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-05-15T13:00:33.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:427: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-05-15T13:00:33.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-15T13:00:33.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-15T13:00:33.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-15T13:00:33.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-15T13:00:33.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-15T13:00:33.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-15T13:00:33.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-05-15T13:00:33.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-15T13:00:33.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-15T13:00:33.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-15T13:00:33.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-05-15T13:00:33.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-15T13:00:33.884 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-05-15T13:00:33.884 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T13:00:33.884 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-05-15T13:00:33.884 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-05-15T13:00:33.884 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T13:00:34.338 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T13:00:34.338 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T13:00:34.339 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:00:34.339 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T13:00:34.339 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T13:00:34.339 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:00:34.339 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T13:00:34.339 INFO:tasks.workunit.client.0.smithi005.stderr:7' 2024-05-15T13:00:34.339 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-05-15T13:00:34.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-05-15T13:00:34.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-15T13:00:34.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-05-15T13:00:34.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-15T13:00:34.343 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-05-15T13:00:34.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-15T13:00:34.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-05-15T13:00:34.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-15T13:00:34.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-05-15T13:00:34.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-15T13:00:34.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-15T13:00:34.343 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-15T13:00:34.343 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-05-15T13:00:34.343 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-15T13:00:34.344 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-15T13:00:34.344 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:00:34.344 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:00:34.344 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:00:34.344 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.3.asok 2024-05-15T13:00:34.344 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.80702/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-05-15T13:00:34.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-15T13:00:34.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-15T13:00:34.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-15T13:00:34.455 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-05-15T13:00:34.456 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-15T13:00:34.456 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-15T13:00:34.456 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:00:34.456 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:00:34.456 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:00:34.456 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.3.asok 2024-05-15T13:00:34.456 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-15T13:00:34.457 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-05-15T13:00:34.574 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:430: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 3 2024-05-15T13:00:34.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-15T13:00:34.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-15T13:00:34.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-15T13:00:34.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-15T13:00:34.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-15T13:00:34.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-15T13:00:34.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-05-15T13:00:34.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-15T13:00:34.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-15T13:00:34.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-15T13:00:34.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=3 2024-05-15T13:00:34.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-15T13:00:34.575 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-05-15T13:00:34.576 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T13:00:34.576 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-05-15T13:00:34.576 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-05-15T13:00:34.576 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T13:00:34.980 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T13:00:34.981 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T13:00:34.981 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:00:34.981 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T13:00:34.981 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T13:00:34.981 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:00:34.981 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T13:00:34.981 INFO:tasks.workunit.client.0.smithi005.stderr:7' 2024-05-15T13:00:34.981 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-05-15T13:00:34.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-05-15T13:00:34.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-15T13:00:34.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-05-15T13:00:34.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-15T13:00:34.982 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/0/type 2024-05-15T13:00:34.983 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-15T13:00:34.983 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-05-15T13:00:34.983 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-15T13:00:34.983 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-05-15T13:00:34.983 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-15T13:00:34.983 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-15T13:00:34.984 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-15T13:00:34.984 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-05-15T13:00:34.984 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T13:00:34.984 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T13:00:34.984 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:00:34.984 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:00:34.985 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:00:34.985 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.0.asok 2024-05-15T13:00:34.985 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.80702/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-05-15T13:00:35.111 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-15T13:00:35.111 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-15T13:00:35.111 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-15T13:00:35.112 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-05-15T13:00:35.112 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T13:00:35.113 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T13:00:35.113 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:00:35.113 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:00:35.114 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:00:35.114 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.0.asok 2024-05-15T13:00:35.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-15T13:00:35.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-osd.0.asok injectdataerr pool-jerasure myobject 3 2024-05-15T13:00:35.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:431: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 4 2024-05-15T13:00:35.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-15T13:00:35.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-15T13:00:35.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-15T13:00:35.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-15T13:00:35.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-15T13:00:35.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-15T13:00:35.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-05-15T13:00:35.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-15T13:00:35.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-15T13:00:35.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-15T13:00:35.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=4 2024-05-15T13:00:35.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-15T13:00:35.238 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-05-15T13:00:35.238 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T13:00:35.238 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-05-15T13:00:35.238 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-05-15T13:00:35.238 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T13:00:35.648 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T13:00:35.648 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T13:00:35.649 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:00:35.649 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T13:00:35.649 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T13:00:35.649 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:00:35.649 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T13:00:35.649 INFO:tasks.workunit.client.0.smithi005.stderr:7' 2024-05-15T13:00:35.649 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-05-15T13:00:35.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-05-15T13:00:35.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-15T13:00:35.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=6 2024-05-15T13:00:35.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-15T13:00:35.649 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/6/type 2024-05-15T13:00:35.650 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-15T13:00:35.650 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 6 bluestore_debug_inject_read_err true 2024-05-15T13:00:35.650 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-15T13:00:35.650 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=6 2024-05-15T13:00:35.650 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-15T13:00:35.650 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-15T13:00:35.650 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-15T13:00:35.651 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.6 2024-05-15T13:00:35.651 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.6 2024-05-15T13:00:35.652 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.6 ']' 2024-05-15T13:00:35.652 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:00:35.652 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:00:35.652 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:00:35.652 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.6.asok 2024-05-15T13:00:35.652 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.80702/ceph-osd.6.asok config set bluestore_debug_inject_read_err true 2024-05-15T13:00:35.773 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-15T13:00:35.774 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-15T13:00:35.774 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-15T13:00:35.775 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.6 2024-05-15T13:00:35.775 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.6 2024-05-15T13:00:35.775 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.6 ']' 2024-05-15T13:00:35.776 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:00:35.776 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:00:35.776 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:00:35.776 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.6.asok 2024-05-15T13:00:35.776 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-15T13:00:35.776 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-osd.6.asok injectdataerr pool-jerasure myobject 4 2024-05-15T13:00:35.897 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: get_osds pool-jerasure myobject 2024-05-15T13:00:35.898 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T13:00:35.898 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-05-15T13:00:35.898 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-05-15T13:00:35.898 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T13:00:36.321 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T13:00:36.321 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T13:00:36.321 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:00:36.321 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T13:00:36.321 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T13:00:36.321 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:00:36.321 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T13:00:36.321 INFO:tasks.workunit.client.0.smithi005.stderr:7' 2024-05-15T13:00:36.322 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-05-15T13:00:36.322 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-05-15T13:00:36.322 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: local -a initial_osds 2024-05-15T13:00:36.322 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:434: TEST_ec_recovery_multiple_errors: local last_osd=7 2024-05-15T13:00:36.322 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:436: TEST_ec_recovery_multiple_errors: kill_daemons td/test-erasure-eio TERM osd.7 2024-05-15T13:00:36.322 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:00:36.323 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:00:36.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:00:36.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:00:36.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:00:36.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:00:36.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:437: TEST_ec_recovery_multiple_errors: ceph osd down 7 2024-05-15T13:00:37.699 INFO:tasks.workunit.client.0.smithi005.stderr:osd.7 is already down. 2024-05-15T13:00:37.717 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:438: TEST_ec_recovery_multiple_errors: ceph osd out 7 2024-05-15T13:00:38.444 INFO:tasks.workunit.client.0.smithi005.stderr:osd.7 is already out. 2024-05-15T13:00:38.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:441: TEST_ec_recovery_multiple_errors: wait_for_clean 2024-05-15T13:00:38.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:00:38.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:00:38.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:00:38.463 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:00:38.464 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:00:38.464 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:00:38.465 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:00:38.465 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:00:38.465 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:00:38.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:00:38.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:00:38.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:00:38.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:00:38.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:00:38.585 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:00:39.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:00:39.052 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:00:39.052 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:00:39.052 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T13:00:39.052 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T13:00:39.052 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T13:00:39.052 INFO:tasks.workunit.client.0.smithi005.stderr:6 2024-05-15T13:00:39.052 INFO:tasks.workunit.client.0.smithi005.stderr:7 2024-05-15T13:00:39.052 INFO:tasks.workunit.client.0.smithi005.stderr:8' 2024-05-15T13:00:39.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:00:39.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:00:39.053 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:00:39.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803851 2024-05-15T13:00:39.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803851 2024-05-15T13:00:39.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851' 2024-05-15T13:00:39.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:00:39.244 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:00:39.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574917 2024-05-15T13:00:39.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574917 2024-05-15T13:00:39.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851 1-55834574917' 2024-05-15T13:00:39.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:00:39.403 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:00:39.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378685 2024-05-15T13:00:39.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378685 2024-05-15T13:00:39.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851 1-55834574917 2-81604378685' 2024-05-15T13:00:39.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:00:39.558 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:00:39.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084341 2024-05-15T13:00:39.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084341 2024-05-15T13:00:39.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851 1-55834574917 2-81604378685 3-120259084341' 2024-05-15T13:00:39.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:00:39.709 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T13:00:39.862 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659177 2024-05-15T13:00:39.862 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659177 2024-05-15T13:00:39.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851 1-55834574917 2-81604378685 3-120259084341 4-176093659177' 2024-05-15T13:00:39.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:00:39.863 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-15T13:00:40.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=219043332132 2024-05-15T13:00:40.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 219043332132 2024-05-15T13:00:40.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851 1-55834574917 2-81604378685 3-120259084341 4-176093659177 5-219043332132' 2024-05-15T13:00:40.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:00:40.026 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-15T13:00:40.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005085 2024-05-15T13:00:40.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005085 2024-05-15T13:00:40.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851 1-55834574917 2-81604378685 3-120259084341 4-176093659177 5-219043332132 6-261993005085' 2024-05-15T13:00:40.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:00:40.186 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-15T13:00:40.407 INFO:tasks.workunit.client.0.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-05-15T13:00:40.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-05-15T13:00:40.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-05-15T13:00:40.410 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-05-15T13:00:40.410 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:00:40.410 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-15T13:00:40.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350987 2024-05-15T13:00:40.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350987 2024-05-15T13:00:40.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851 1-55834574917 2-81604378685 3-120259084341 4-176093659177 5-219043332132 6-261993005085 8-347892350987' 2024-05-15T13:00:40.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:00:40.696 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803851 2024-05-15T13:00:40.696 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:00:40.697 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:00:40.698 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803851 2024-05-15T13:00:40.698 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:00:40.699 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803851 2024-05-15T13:00:40.699 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 25769803851 2024-05-15T13:00:40.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803851' 2024-05-15T13:00:40.700 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:00:41.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803851 -lt 25769803851 2024-05-15T13:00:41.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:00:41.262 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574917 2024-05-15T13:00:41.262 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:00:41.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:00:41.264 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574917 2024-05-15T13:00:41.265 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:00:41.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574917 2024-05-15T13:00:41.266 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 55834574917 2024-05-15T13:00:41.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574917' 2024-05-15T13:00:41.267 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:00:41.856 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574917 -lt 55834574917 2024-05-15T13:00:41.856 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:00:41.857 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378685 2024-05-15T13:00:41.857 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:00:41.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:00:41.860 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378685 2024-05-15T13:00:41.860 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:00:41.861 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378685 2024-05-15T13:00:41.862 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 81604378685 2024-05-15T13:00:41.862 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378685' 2024-05-15T13:00:41.862 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:00:42.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378686 -lt 81604378685 2024-05-15T13:00:42.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:00:42.281 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:00:42.281 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084341 2024-05-15T13:00:42.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:00:42.283 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084341 2024-05-15T13:00:42.283 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:00:42.284 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084341 2024-05-15T13:00:42.284 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 120259084341 2024-05-15T13:00:42.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084341' 2024-05-15T13:00:42.285 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:00:42.786 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084341 -lt 120259084341 2024-05-15T13:00:42.786 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:00:42.787 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-176093659177 2024-05-15T13:00:42.787 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:00:42.789 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T13:00:42.790 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-176093659177 2024-05-15T13:00:42.790 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:00:42.791 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659177 2024-05-15T13:00:42.791 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.4 seq 176093659177 2024-05-15T13:00:42.791 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 176093659177' 2024-05-15T13:00:42.792 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T13:00:43.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659177 -lt 176093659177 2024-05-15T13:00:43.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:00:43.215 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-219043332132 2024-05-15T13:00:43.215 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:00:43.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-15T13:00:43.218 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-219043332132 2024-05-15T13:00:43.218 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:00:43.219 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=219043332132 2024-05-15T13:00:43.220 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.5 seq 219043332132 2024-05-15T13:00:43.220 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 219043332132' 2024-05-15T13:00:43.220 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-15T13:00:43.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 219043332133 -lt 219043332132 2024-05-15T13:00:43.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:00:43.641 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-261993005085 2024-05-15T13:00:43.641 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:00:43.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-15T13:00:43.644 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-261993005085 2024-05-15T13:00:43.644 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:00:43.646 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005085 2024-05-15T13:00:43.646 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 261993005085' 2024-05-15T13:00:43.646 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.6 seq 261993005085 2024-05-15T13:00:43.646 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-15T13:00:44.089 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005086 -lt 261993005085 2024-05-15T13:00:44.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:00:44.090 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-347892350987 2024-05-15T13:00:44.090 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:00:44.091 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-15T13:00:44.092 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-347892350987 2024-05-15T13:00:44.092 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:00:44.094 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350987 2024-05-15T13:00:44.094 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 347892350987' 2024-05-15T13:00:44.094 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.8 seq 347892350987 2024-05-15T13:00:44.094 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-15T13:00:44.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350988 -lt 347892350987 2024-05-15T13:00:44.537 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:00:44.538 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:00:44.538 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:00:45.210 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:00:45.210 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:00:45.210 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:00:45.210 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:00:45.210 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:00:45.211 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:00:45.211 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:00:45.211 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:00:45.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:00:45.624 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:00:45.624 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:00:45.625 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:00:46.192 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:00:46.192 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:00:46.192 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:00:46.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:443: TEST_ec_recovery_multiple_errors: rados_get td/test-erasure-eio pool-jerasure myobject 2024-05-15T13:00:46.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-15T13:00:46.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-15T13:00:46.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-05-15T13:00:46.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-15T13:00:46.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-15T13:00:46.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-05-15T13:00:46.250 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-15T13:00:46.250 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-15T13:00:46.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:445: TEST_ec_recovery_multiple_errors: delete_erasure_coded_pool pool-jerasure 2024-05-15T13:00:46.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-15T13:00:46.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-15T13:00:46.768 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-jerasure' does not exist 2024-05-15T13:00:46.784 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-15T13:00:47.892 INFO:tasks.workunit.client.0.smithi005.stderr:erasure-code-profile myprofile does not exist 2024-05-15T13:00:47.911 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-15T13:00:47.911 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-15T13:00:47.911 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T13:00:47.911 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-15T13:00:47.912 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:00:47.912 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:00:47.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:00:47.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:00:47.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:00:48.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:00:48.229 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T13:00:48.230 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T13:00:48.230 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T13:00:48.232 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T13:00:48.232 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T13:00:48.232 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T13:00:48.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:00:48.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T13:00:48.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T13:00:48.234 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:00:48.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:00:48.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T13:00:48.237 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-15T13:00:48.290 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T13:00:48.290 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:00:48.290 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:00:48.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80702 2024-05-15T13:00:48.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T13:00:48.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T13:00:48.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-15T13:00:48.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-15T13:00:48.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-15T13:00:48.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-15T13:00:48.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-15T13:00:48.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T13:00:48.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-15T13:00:48.293 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:00:48.293 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:00:48.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:00:48.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:00:48.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:00:48.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:00:48.295 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T13:00:48.296 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T13:00:48.296 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T13:00:48.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T13:00:48.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T13:00:48.298 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T13:00:48.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:00:48.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T13:00:48.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T13:00:48.299 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:00:48.300 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:00:48.301 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T13:00:48.302 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-15T13:00:48.303 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T13:00:48.303 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:00:48.303 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:00:48.303 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80702 2024-05-15T13:00:48.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T13:00:48.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T13:00:48.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-15T13:00:48.306 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T13:00:48.307 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:00:48.307 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:00:48.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.80702 2024-05-15T13:00:48.309 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T13:00:48.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-15T13:00:48.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T13:00:48.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-15T13:00:48.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-15T13:00:48.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-15T13:00:48.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T13:00:48.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T13:00:48.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T13:00:48.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-15T13:00:48.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-15T13:00:48.720 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T13:00:48.720 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:00:48.720 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:00:48.721 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:00:48.721 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:00:48.721 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:00:48.721 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:00:48.721 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-15T13:00:48.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T13:00:48.939 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T13:00:48.939 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T13:00:48.939 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T13:00:48.939 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T13:00:48.939 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T13:00:48.940 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T13:00:48.940 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T13:00:48.940 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T13:00:48.941 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:00:48.941 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:00:48.941 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:00:48.941 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T13:00:48.941 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T13:00:48.941 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.80702/ceph-mon.a.asok config get fsid 2024-05-15T13:00:49.058 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T13:00:49.058 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T13:00:49.059 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T13:00:49.059 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T13:00:49.060 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T13:00:49.060 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T13:00:49.060 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T13:00:49.060 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T13:00:49.060 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:00:49.060 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:00:49.060 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:00:49.060 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T13:00:49.060 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T13:00:49.061 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.80702/ceph-mon.a.asok config get mon_host 2024-05-15T13:00:49.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-15T13:00:49.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-15T13:00:49.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T13:00:49.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T13:00:49.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T13:00:49.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-15T13:00:49.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T13:00:50.125 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T13:00:50.126 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:00:50.126 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:00:50.126 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:00:50.126 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:00:50.126 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:00:50.126 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:00:50.127 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T13:00:50.127 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T13:00:50.187 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-15T13:00:50.187 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-15T13:00:50.526 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd' already exists 2024-05-15T13:00:50.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T13:00:51.545 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-15T13:00:51.546 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T13:00:51.546 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T13:00:51.546 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:00:51.546 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:00:51.547 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:00:51.547 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T13:00:51.547 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-15T13:00:51.547 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-mon.a.asok log flush 2024-05-15T13:00:51.649 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T13:00:51.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-15T13:00:51.663 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T13:00:48.779+0000 7f4ce3366c80 0 load: jerasure load: lrc 2024-05-15T13:00:51.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects td/test-erasure-eio 2024-05-15T13:00:51.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:451: TEST_ec_recovery_multiple_objects: local dir=td/test-erasure-eio 2024-05-15T13:00:51.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:452: TEST_ec_recovery_multiple_objects: local objname=myobject 2024-05-15T13:00:51.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:454: TEST_ec_recovery_multiple_objects: ORIG_ARGS='--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:00:51.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:455: TEST_ec_recovery_multiple_objects: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-15T13:00:51.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:456: TEST_ec_recovery_multiple_objects: setup_osds 7 2024-05-15T13:00:51.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-05-15T13:00:51.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-15T13:00:51.665 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-05-15T13:00:51.666 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-05-15T13:00:51.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:00:51.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-15T13:00:51.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:00:51.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:00:51.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T13:00:51.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:00:51.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-15T13:00:51.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-15T13:00:51.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:00:51.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:00:51.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:00:51.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-15T13:00:51.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-15T13:00:51.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:00:51.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:00:51.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:00:51.672 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:00:51.672 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:00:51.672 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:00:51.672 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:00:51.673 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:00:51.673 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:00:51.673 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:00:51.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:00:51.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:00:51.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:00:51.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:00:51.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:00:51.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:00:51.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:00:51.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:00:51.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:00:51.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:00:51.674 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:00:51.674 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:00:51.674 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-15T13:00:51.674 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:00:51.674 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=327ac897-94e0-416a-8a73-354947dd4000 2024-05-15T13:00:51.675 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 327ac897-94e0-416a-8a73-354947dd4000 2024-05-15T13:00:51.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 327ac897-94e0-416a-8a73-354947dd4000' 2024-05-15T13:00:51.675 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:00:51.693 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQADskRm49UYKhAA0nBrKFcL/mH2KSUZWhdR0Q== 2024-05-15T13:00:51.693 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQADskRm49UYKhAA0nBrKFcL/mH2KSUZWhdR0Q=="}' 2024-05-15T13:00:51.693 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 327ac897-94e0-416a-8a73-354947dd4000 -i td/test-erasure-eio/0/new.json 2024-05-15T13:00:52.036 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:00:52.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-15T13:00:52.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQADskRm49UYKhAA0nBrKFcL/mH2KSUZWhdR0Q== --osd-uuid 327ac897-94e0-416a-8a73-354947dd4000 2024-05-15T13:00:52.098 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:00:52.107+0000 7f75fc59d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:00:52.104 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:00:52.115+0000 7f75fc59d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:00:52.105 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:00:52.115+0000 7f75fc59d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:00:52.106 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:00:52.115+0000 7f75fc59d7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T13:00:52.106 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:00:52.115+0000 7f75fc59d7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T13:00:52.106 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:00:52.115+0000 7f75fc59d7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T13:00:52.135 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:00:52.147+0000 7f75fc59d7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-15T13:00:58.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-15T13:00:58.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:00:58.873 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T13:00:58.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T13:00:58.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:00:59.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T13:00:59.435 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T13:00:59.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:00:59.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:00:59.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:00:59.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:00:59.473 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:00:59.482+0000 7febe988e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:00:59.501 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:00:59.510+0000 7febe988e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:00:59.503 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:00:59.514+0000 7febe988e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:00:59.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T13:00:59.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:00:59.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T13:00:59.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:00:59.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:00:59.875 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:00:59.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:00:59.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:00:59.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:00:59.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:01:00.064 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:01:00.074+0000 7febe988e7c0 -1 Falling back to public interface 2024-05-15T13:01:00.301 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:01.303 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:01.303 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:01.303 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:01:01.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:01:01.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:01.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:01:01.736 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:02.691 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:01:02.702+0000 7febe988e7c0 -1 osd.0 0 log_to_monitors true 2024-05-15T13:01:02.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:02.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:02.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:01:02.739 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:01:02.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:02.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:01:03.452 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:04.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:04.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:04.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:01:04.455 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:01:04.455 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:04.455 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:01:04.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:05.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:05.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:05.920 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:01:05.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:01:05.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:05.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:01:06.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:07.398 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:07.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:07.399 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:01:07.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:01:07.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:07.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:01:07.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:08.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:08.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:08.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:01:08.876 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:01:08.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:08.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:01:09.324 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:10.326 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:10.326 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:10.326 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:01:10.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:01:10.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:10.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:01:10.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:11.870 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:11.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:11.871 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:01:11.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:01:11.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:11.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:01:12.316 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:13.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:13.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:13.318 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:01:13.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:01:13.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:13.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:01:13.946 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:14.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:14.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:14.950 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:01:14.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:01:14.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:14.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:01:15.509 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:16.511 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:16.511 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:16.511 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:01:16.512 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:01:16.512 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:16.512 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:01:17.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:18.197 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:18.197 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:18.197 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:01:18.198 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:01:18.198 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:18.198 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:01:19.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:20.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:20.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:20.009 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:01:20.010 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:01:20.010 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:20.010 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:01:20.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:21.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:21.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:21.796 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:01:21.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:01:21.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:21.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:01:22.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:23.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:23.692 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:23.692 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:01:23.693 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:01:23.693 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:23.693 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:01:24.857 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:25.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:25.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:25.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:01:25.859 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:01:25.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:25.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:01:26.552 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:27.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:27.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:27.555 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:01:27.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:01:27.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:27.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:01:28.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:29.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:29.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:29.084 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:01:29.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:01:29.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:29.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:01:29.628 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3020551110,v1:127.0.0.1:6803/3020551110] [v2:127.0.0.1:6804/3020551110,v1:127.0.0.1:6805/3020551110] exists,up 327ac897-94e0-416a-8a73-354947dd4000 2024-05-15T13:01:29.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:01:29.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:01:29.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:01:29.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:01:29.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-15T13:01:29.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:01:29.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:01:29.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-15T13:01:29.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:01:29.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-15T13:01:29.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-15T13:01:29.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:01:29.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:01:29.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:01:29.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-15T13:01:29.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-15T13:01:29.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:01:29.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:01:29.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:01:29.633 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:01:29.633 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:01:29.633 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:01:29.633 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:01:29.633 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:01:29.633 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:01:29.633 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:01:29.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:01:29.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:01:29.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:01:29.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:01:29.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:01:29.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:01:29.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:01:29.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:01:29.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:01:29.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:01:29.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:01:29.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:01:29.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-15T13:01:29.635 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:01:29.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a94c5554-8f80-44a1-9456-14925eb0dc22 2024-05-15T13:01:29.635 INFO:tasks.workunit.client.0.smithi005.stdout:add osd1 a94c5554-8f80-44a1-9456-14925eb0dc22 2024-05-15T13:01:29.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a94c5554-8f80-44a1-9456-14925eb0dc22' 2024-05-15T13:01:29.636 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:01:29.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQApskRmCFwcKBAAppIOxe3c72kaaZWMhHQiVQ== 2024-05-15T13:01:29.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQApskRmCFwcKBAAppIOxe3c72kaaZWMhHQiVQ=="}' 2024-05-15T13:01:29.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a94c5554-8f80-44a1-9456-14925eb0dc22 -i td/test-erasure-eio/1/new.json 2024-05-15T13:01:30.150 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:01:30.166 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-15T13:01:30.167 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQApskRmCFwcKBAAppIOxe3c72kaaZWMhHQiVQ== --osd-uuid a94c5554-8f80-44a1-9456-14925eb0dc22 2024-05-15T13:01:30.216 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:01:30.226+0000 7fb51ccc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:01:30.218 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:01:30.230+0000 7fb51ccc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:01:30.219 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:01:30.230+0000 7fb51ccc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:01:30.220 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:01:30.230+0000 7fb51ccc47c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T13:01:30.220 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:01:30.230+0000 7fb51ccc47c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T13:01:30.220 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:01:30.230+0000 7fb51ccc47c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T13:01:30.252 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:01:30.262+0000 7fb51ccc47c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-15T13:01:37.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-15T13:01:37.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:01:37.078 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd1 key to auth repository 2024-05-15T13:01:37.078 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-15T13:01:37.078 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:01:37.644 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.1 2024-05-15T13:01:37.644 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-15T13:01:37.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:01:37.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:01:37.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:01:37.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:01:37.682 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:01:37.689+0000 7fb8b1fca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:01:37.708 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:01:37.717+0000 7fb8b1fca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:01:37.710 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:01:37.717+0000 7fb8b1fca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:01:38.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-15T13:01:38.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:01:38.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-15T13:01:38.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:01:38.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:01:38.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:38.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:01:38.115 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:01:38.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:38.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:01:38.267 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:01:38.277+0000 7fb8b1fca7c0 -1 Falling back to public interface 2024-05-15T13:01:38.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:39.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:39.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:39.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:01:39.607 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:01:39.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:39.608 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:01:40.031 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:40.955 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:01:40.965+0000 7fb8b1fca7c0 -1 osd.1 0 log_to_monitors true 2024-05-15T13:01:41.033 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:41.033 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:41.033 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:01:41.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:01:41.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:41.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:01:41.946 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:42.948 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:42.948 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:42.949 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:01:42.949 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:01:42.949 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:42.949 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:01:43.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:44.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:44.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:44.593 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:01:44.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:01:44.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:44.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:01:45.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:46.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:46.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:46.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:01:46.059 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:01:46.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:46.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:01:46.528 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:47.530 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:47.530 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:47.530 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:01:47.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:01:47.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:47.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:01:48.091 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:49.093 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:49.094 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:49.094 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:01:49.094 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:01:49.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:49.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:01:49.609 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:50.610 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:50.610 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:50.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:01:50.611 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:01:50.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:50.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:01:51.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:52.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:52.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:52.272 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:01:52.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:01:52.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:52.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:01:52.719 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:53.721 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:53.721 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:53.721 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:01:53.722 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:01:53.722 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:53.722 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:01:54.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:55.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:55.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:55.189 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:01:55.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:01:55.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:55.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:01:56.010 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:57.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:57.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:57.012 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:01:57.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:01:57.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:57.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:01:57.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:01:58.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:01:58.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:01:58.632 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:01:58.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:01:58.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:01:58.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:01:59.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:00.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:00.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:00.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:02:00.445 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:02:00.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:00.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:02:01.345 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:02.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:02.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:02.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:02:02.347 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:02:02.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:02.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:02:03.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:04.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:04.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:04.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:02:04.217 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:02:04.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:04.218 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:02:04.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:05.958 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:05.958 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:05.959 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:02:05.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:02:05.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:05.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:02:06.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:07.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:07.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:07.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:02:07.630 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:02:07.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:07.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:02:08.354 INFO:tasks.workunit.client.0.smithi005.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1354140539,v1:127.0.0.1:6811/1354140539] [v2:127.0.0.1:6812/1354140539,v1:127.0.0.1:6813/1354140539] exists,up a94c5554-8f80-44a1-9456-14925eb0dc22 2024-05-15T13:02:08.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:02:08.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:02:08.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:02:08.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:02:08.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-15T13:02:08.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:02:08.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:02:08.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-15T13:02:08.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:02:08.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-15T13:02:08.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-15T13:02:08.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:02:08.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:02:08.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:02:08.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-15T13:02:08.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-15T13:02:08.359 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:02:08.359 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:02:08.359 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:02:08.359 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:02:08.359 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:02:08.359 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:02:08.359 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:02:08.359 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:02:08.359 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:02:08.361 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:02:08.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:02:08.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:02:08.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:02:08.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:02:08.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:02:08.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:02:08.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:02:08.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:02:08.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:02:08.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:02:08.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:02:08.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:02:08.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-15T13:02:08.362 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:02:08.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0e0bad0d-a7f6-40eb-a6e1-328f62812453 2024-05-15T13:02:08.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 0e0bad0d-a7f6-40eb-a6e1-328f62812453' 2024-05-15T13:02:08.363 INFO:tasks.workunit.client.0.smithi005.stdout:add osd2 0e0bad0d-a7f6-40eb-a6e1-328f62812453 2024-05-15T13:02:08.363 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:02:08.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBQskRmuKb8FxAAqyI82xiao9fsMGoPH/L9Xw== 2024-05-15T13:02:08.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBQskRmuKb8FxAAqyI82xiao9fsMGoPH/L9Xw=="}' 2024-05-15T13:02:08.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0e0bad0d-a7f6-40eb-a6e1-328f62812453 -i td/test-erasure-eio/2/new.json 2024-05-15T13:02:09.143 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:02:09.161 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-15T13:02:09.162 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBQskRmuKb8FxAAqyI82xiao9fsMGoPH/L9Xw== --osd-uuid 0e0bad0d-a7f6-40eb-a6e1-328f62812453 2024-05-15T13:02:09.197 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:09.205+0000 7f74629397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:02:09.253 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:09.261+0000 7f74629397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:02:09.255 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:09.265+0000 7f74629397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:02:09.255 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:09.265+0000 7f74629397c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T13:02:09.256 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:09.265+0000 7f74629397c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T13:02:09.256 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:09.265+0000 7f74629397c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T13:02:09.377 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:09.385+0000 7f74629397c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-15T13:02:16.383 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-15T13:02:16.392 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:02:16.392 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd2 key to auth repository 2024-05-15T13:02:16.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-15T13:02:16.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:02:16.954 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.2 2024-05-15T13:02:16.955 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-15T13:02:16.955 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:02:16.955 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:02:16.955 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:02:16.955 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:02:16.992 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:17.000+0000 7f7c4e2927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:02:17.028 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:17.036+0000 7f7c4e2927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:02:17.029 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:17.036+0000 7f7c4e2927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:02:17.581 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:17.588+0000 7f7c4e2927c0 -1 Falling back to public interface 2024-05-15T13:02:17.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-15T13:02:17.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:02:17.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-15T13:02:17.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:02:17.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:02:17.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:17.606 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:02:17.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:02:17.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:17.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:02:18.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:19.044 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:19.044 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:02:19.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:19.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:02:19.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:19.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:02:19.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:20.324 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:20.332+0000 7f7c4e2927c0 -1 osd.2 0 log_to_monitors true 2024-05-15T13:02:20.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:20.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:20.487 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:02:20.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:02:20.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:20.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:02:21.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:22.106 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:22.106 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:22.106 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:02:22.107 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:02:22.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:22.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:02:22.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:23.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:23.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:23.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:02:23.640 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:02:23.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:23.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:02:24.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:25.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:25.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:25.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:02:25.182 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:02:25.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:25.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:02:25.697 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:26.699 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:26.699 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:26.700 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:02:26.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:02:26.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:26.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:02:27.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:28.316 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:28.316 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:28.316 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:02:28.316 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:02:28.316 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:28.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:02:28.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:29.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:29.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:29.814 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:02:29.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:02:29.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:29.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:02:30.359 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:31.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:31.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:31.361 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:02:31.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:02:31.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:31.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:02:31.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:32.853 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:32.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:32.854 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:02:32.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:02:32.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:32.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:02:33.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:34.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:34.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:34.508 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:02:34.508 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:02:34.508 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:34.508 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:02:35.203 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:36.205 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:36.205 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:36.205 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:02:36.206 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:02:36.206 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:36.206 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:02:36.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:37.982 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:37.982 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:37.982 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:02:37.983 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:02:37.983 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:37.983 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:02:38.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:39.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:39.677 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:39.677 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:02:39.677 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:02:39.677 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:39.678 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:02:40.595 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:41.596 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:41.597 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:41.597 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:02:41.597 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:02:41.597 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:41.597 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:02:42.650 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:43.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:43.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:43.652 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:02:43.653 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:02:43.653 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:43.653 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:02:44.771 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:45.773 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:45.773 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:45.773 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:02:45.773 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:02:45.774 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:45.774 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:02:46.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:47.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:47.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:47.377 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:02:47.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:02:47.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:47.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:02:47.952 INFO:tasks.workunit.client.0.smithi005.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3625582943,v1:127.0.0.1:6819/3625582943] [v2:127.0.0.1:6820/3625582943,v1:127.0.0.1:6821/3625582943] exists,up 0e0bad0d-a7f6-40eb-a6e1-328f62812453 2024-05-15T13:02:47.958 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:02:47.958 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:02:47.958 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:02:47.958 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:02:47.958 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-15T13:02:47.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:02:47.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:02:47.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-15T13:02:47.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:02:47.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-15T13:02:47.959 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-15T13:02:47.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:02:47.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:02:47.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:02:47.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-15T13:02:47.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-15T13:02:47.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:02:47.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:02:47.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:02:47.961 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:02:47.961 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:02:47.961 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:02:47.961 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:02:47.962 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:02:47.962 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:02:47.962 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:02:47.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:02:47.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:02:47.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:02:47.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:02:47.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:02:47.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:02:47.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:02:47.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:02:47.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:02:47.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:02:47.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:02:47.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:02:47.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-15T13:02:47.964 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:02:47.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ef7b587d-0a11-40a3-97f7-888f92ccefa7 2024-05-15T13:02:47.965 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 ef7b587d-0a11-40a3-97f7-888f92ccefa7' 2024-05-15T13:02:47.965 INFO:tasks.workunit.client.0.smithi005.stdout:add osd3 ef7b587d-0a11-40a3-97f7-888f92ccefa7 2024-05-15T13:02:47.966 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:02:47.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB3skRmvCB2OxAARfVNDPqMh/NjJYOe3ZmrLw== 2024-05-15T13:02:47.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB3skRmvCB2OxAARfVNDPqMh/NjJYOe3ZmrLw=="}' 2024-05-15T13:02:47.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ef7b587d-0a11-40a3-97f7-888f92ccefa7 -i td/test-erasure-eio/3/new.json 2024-05-15T13:02:49.250 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:02:49.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-15T13:02:49.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB3skRmvCB2OxAARfVNDPqMh/NjJYOe3ZmrLw== --osd-uuid ef7b587d-0a11-40a3-97f7-888f92ccefa7 2024-05-15T13:02:49.315 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:49.328+0000 7fcafba0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:02:49.359 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:49.372+0000 7fcafba0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:02:49.361 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:49.372+0000 7fcafba0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:02:49.361 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:49.372+0000 7fcafba0e7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T13:02:49.361 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:49.372+0000 7fcafba0e7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T13:02:49.361 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:49.372+0000 7fcafba0e7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T13:02:49.551 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:49.564+0000 7fcafba0e7c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-15T13:02:56.588 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-15T13:02:56.588 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:02:56.590 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd3 key to auth repository 2024-05-15T13:02:56.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-15T13:02:56.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:02:57.199 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.3 2024-05-15T13:02:57.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-15T13:02:57.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:02:57.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:02:57.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:02:57.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:02:57.238 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:57.248+0000 7fd21f7037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:02:57.249 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:57.260+0000 7fd21f7037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:02:57.250 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:57.260+0000 7fd21f7037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:02:57.621 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-15T13:02:57.621 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:02:57.621 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-15T13:02:57.621 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:02:57.621 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:02:57.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:02:57.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:57.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:02:57.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:57.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:02:57.801 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:02:57.812+0000 7fd21f7037c0 -1 Falling back to public interface 2024-05-15T13:02:58.091 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:02:59.093 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:02:59.094 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:02:59.094 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:02:59.094 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:02:59.094 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:02:59.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:02:59.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:00.432 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:03:00.443+0000 7fd21f7037c0 -1 osd.3 0 log_to_monitors true 2024-05-15T13:03:00.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:00.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:00.541 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:03:00.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:03:00.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:00.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:03:01.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:02.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:02.395 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:02.395 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:03:02.395 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:03:02.395 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:02.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:03:03.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:04.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:04.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:04.357 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:03:04.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:03:04.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:04.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:03:04.793 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:05.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:05.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:05.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:03:05.796 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:03:05.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:05.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:03:06.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:07.401 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:07.401 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:07.401 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:03:07.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:03:07.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:07.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:03:07.974 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:08.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:08.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:08.976 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:03:08.977 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:03:08.977 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:08.977 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:03:09.553 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:10.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:10.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:10.555 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:03:10.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:03:10.556 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:10.556 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:03:11.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:12.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:12.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:12.007 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:03:12.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:03:12.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:12.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:03:12.482 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:13.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:13.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:13.484 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:03:13.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:03:13.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:13.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:03:13.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:14.929 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:14.929 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:14.929 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:03:14.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:03:14.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:14.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:03:15.474 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:16.475 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:16.476 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:16.476 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:03:16.476 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:03:16.476 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:16.476 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:03:17.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:18.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:18.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:18.308 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:03:18.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:03:18.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:18.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:03:19.162 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:20.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:20.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:20.164 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:03:20.164 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:03:20.164 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:20.164 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:03:21.028 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:22.030 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:22.030 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:22.031 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:03:22.031 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:03:22.031 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:22.031 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:03:22.832 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:23.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:23.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:23.834 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:03:23.835 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:03:23.835 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:23.835 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:03:25.066 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:26.068 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:26.068 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:26.069 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:03:26.069 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:03:26.069 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:26.069 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:03:26.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:27.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:27.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:27.734 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:03:27.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:03:27.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:27.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:03:28.308 INFO:tasks.workunit.client.0.smithi005.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3175402256,v1:127.0.0.1:6827/3175402256] [v2:127.0.0.1:6828/3175402256,v1:127.0.0.1:6829/3175402256] exists,up ef7b587d-0a11-40a3-97f7-888f92ccefa7 2024-05-15T13:03:28.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:03:28.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:03:28.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:03:28.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:03:28.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-05-15T13:03:28.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:03:28.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:03:28.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-05-15T13:03:28.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:03:28.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-05-15T13:03:28.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-15T13:03:28.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:03:28.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:03:28.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:03:28.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-05-15T13:03:28.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-05-15T13:03:28.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:03:28.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:03:28.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:03:28.314 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:03:28.314 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:03:28.314 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:03:28.314 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:03:28.314 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:03:28.314 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:03:28.314 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:03:28.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:03:28.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:03:28.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:03:28.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:03:28.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:03:28.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:03:28.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:03:28.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:03:28.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:03:28.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:03:28.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:03:28.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:03:28.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-05-15T13:03:28.315 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:03:28.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3edebe4c-c00b-4b5d-b459-7ac9e28d81d0 2024-05-15T13:03:28.316 INFO:tasks.workunit.client.0.smithi005.stdout:add osd4 3edebe4c-c00b-4b5d-b459-7ac9e28d81d0 2024-05-15T13:03:28.316 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 3edebe4c-c00b-4b5d-b459-7ac9e28d81d0' 2024-05-15T13:03:28.316 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:03:28.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCgskRmk+/mFBAALBmZ2QxSWY4/aXuA5WbzhQ== 2024-05-15T13:03:28.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCgskRmk+/mFBAALBmZ2QxSWY4/aXuA5WbzhQ=="}' 2024-05-15T13:03:28.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3edebe4c-c00b-4b5d-b459-7ac9e28d81d0 -i td/test-erasure-eio/4/new.json 2024-05-15T13:03:29.497 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:03:29.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-05-15T13:03:29.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCgskRmk+/mFBAALBmZ2QxSWY4/aXuA5WbzhQ== --osd-uuid 3edebe4c-c00b-4b5d-b459-7ac9e28d81d0 2024-05-15T13:03:29.562 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:03:29.571+0000 7f06ba16a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:03:29.564 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:03:29.575+0000 7f06ba16a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:03:29.565 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:03:29.575+0000 7f06ba16a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:03:29.566 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:03:29.575+0000 7f06ba16a7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T13:03:29.566 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:03:29.575+0000 7f06ba16a7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T13:03:29.566 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:03:29.575+0000 7f06ba16a7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T13:03:29.704 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:03:29.715+0000 7f06ba16a7c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-05-15T13:03:36.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-05-15T13:03:36.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:03:36.879 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd4 key to auth repository 2024-05-15T13:03:36.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-05-15T13:03:36.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:03:37.449 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.4 2024-05-15T13:03:37.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-05-15T13:03:37.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:03:37.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:03:37.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:03:37.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:03:37.486 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:03:37.495+0000 7fcdc00a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:03:37.499 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:03:37.511+0000 7fcdc00a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:03:37.500 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:03:37.511+0000 7fcdc00a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:03:37.901 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-05-15T13:03:37.901 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:03:37.901 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-05-15T13:03:37.901 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:03:37.901 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:03:37.901 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:37.901 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:03:37.902 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:03:37.902 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:37.902 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:03:38.064 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:03:38.075+0000 7fcdc00a57c0 -1 Falling back to public interface 2024-05-15T13:03:38.341 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:39.342 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:39.342 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:39.343 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:03:39.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:03:39.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:39.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:03:39.776 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:40.741 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:03:40.751+0000 7fcdc00a57c0 -1 osd.4 0 log_to_monitors true 2024-05-15T13:03:40.779 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:40.779 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:40.779 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:03:40.779 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:03:40.779 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:40.779 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:03:41.572 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:42.574 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:42.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:42.575 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:03:42.576 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:03:42.576 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:42.576 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:03:43.415 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:44.417 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:44.417 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:44.417 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:03:44.417 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:03:44.418 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:44.418 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:03:45.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:46.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:46.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:46.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:03:46.268 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:03:46.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:46.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:03:46.841 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:47.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:47.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:47.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:03:47.843 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:03:47.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:47.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:03:48.300 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:49.301 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:49.301 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:49.301 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:03:49.302 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:03:49.302 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:49.302 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:03:49.791 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:50.793 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:50.793 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:50.793 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:03:50.793 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:03:50.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:50.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:03:51.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:52.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:52.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:52.310 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:03:52.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:03:52.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:52.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:03:52.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:53.836 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:53.836 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:53.836 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:03:53.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:03:53.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:53.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:03:54.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:55.389 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:55.389 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:55.389 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:03:55.389 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:03:55.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:55.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:03:55.915 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:56.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:56.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:56.916 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:03:56.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:03:56.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:56.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:03:57.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:03:58.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:03:58.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:03:58.503 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:03:58.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:03:58.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:03:58.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:03:59.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:00.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:00.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:00.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:04:00.282 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:04:00.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:00.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:04:01.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:02.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:02.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:02.283 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:04:02.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:04:02.284 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:02.284 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:04:03.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:04.131 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:04.131 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:04.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:04:04.132 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:04:04.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:04.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:04:05.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:06.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:06.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:06.116 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:04:06.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:04:06.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:06.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:04:07.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:08.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:08.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:08.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:04:08.013 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:04:08.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:08.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:04:08.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:09.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:09.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:09.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-15T13:04:09.624 INFO:tasks.workunit.client.0.smithi005.stdout:19 2024-05-15T13:04:09.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:09.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:04:10.371 INFO:tasks.workunit.client.0.smithi005.stdout:osd.4 up in weight 1 up_from 41 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/858250927,v1:127.0.0.1:6835/858250927] [v2:127.0.0.1:6836/858250927,v1:127.0.0.1:6837/858250927] exists,up 3edebe4c-c00b-4b5d-b459-7ac9e28d81d0 2024-05-15T13:04:10.374 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:04:10.374 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:04:10.374 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:04:10.374 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:04:10.374 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-05-15T13:04:10.374 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:04:10.374 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:04:10.374 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-05-15T13:04:10.374 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:04:10.374 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-05-15T13:04:10.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-15T13:04:10.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:04:10.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:04:10.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:04:10.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-05-15T13:04:10.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-05-15T13:04:10.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:04:10.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:04:10.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:04:10.375 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:04:10.375 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:04:10.375 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:04:10.375 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:04:10.375 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:04:10.376 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:04:10.376 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:04:10.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:04:10.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:04:10.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:04:10.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:04:10.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:04:10.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:04:10.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:04:10.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:04:10.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:04:10.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:04:10.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:04:10.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:04:10.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-05-15T13:04:10.378 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:04:10.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ae3b6776-1249-49a1-b629-d10018cefd97 2024-05-15T13:04:10.379 INFO:tasks.workunit.client.0.smithi005.stdout:add osd5 ae3b6776-1249-49a1-b629-d10018cefd97 2024-05-15T13:04:10.379 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 ae3b6776-1249-49a1-b629-d10018cefd97' 2024-05-15T13:04:10.379 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:04:10.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDKskRm6VOLGBAABC6vijBQWu+hKd9PUzX6vw== 2024-05-15T13:04:10.400 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDKskRm6VOLGBAABC6vijBQWu+hKd9PUzX6vw=="}' 2024-05-15T13:04:10.400 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ae3b6776-1249-49a1-b629-d10018cefd97 -i td/test-erasure-eio/5/new.json 2024-05-15T13:04:11.514 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:04:11.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-05-15T13:04:11.539 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKskRm6VOLGBAABC6vijBQWu+hKd9PUzX6vw== --osd-uuid ae3b6776-1249-49a1-b629-d10018cefd97 2024-05-15T13:04:11.580 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:11.590+0000 7fa3fecbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:04:11.641 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:11.650+0000 7fa3fecbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:04:11.642 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:11.650+0000 7fa3fecbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:04:11.643 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:11.654+0000 7fa3fecbb7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-15T13:04:11.643 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:11.654+0000 7fa3fecbb7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-15T13:04:11.643 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:11.654+0000 7fa3fecbb7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-15T13:04:11.823 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:11.834+0000 7fa3fecbb7c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-05-15T13:04:18.648 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-05-15T13:04:18.648 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:04:18.650 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd5 key to auth repository 2024-05-15T13:04:18.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-05-15T13:04:18.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:04:19.232 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.5 2024-05-15T13:04:19.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-05-15T13:04:19.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:04:19.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:04:19.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:04:19.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:04:19.271 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:19.282+0000 7f0af08447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:04:19.282 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:19.290+0000 7f0af08447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:04:19.283 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:19.294+0000 7f0af08447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:04:19.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-05-15T13:04:19.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:04:19.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-05-15T13:04:19.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:04:19.675 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:04:19.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:04:19.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:19.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:04:19.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:19.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:04:19.845 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:19.854+0000 7f0af08447c0 -1 Falling back to public interface 2024-05-15T13:04:20.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:21.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:21.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:21.112 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:04:21.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:04:21.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:21.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:04:21.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:22.538 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:22.546+0000 7f0af08447c0 -1 osd.5 0 log_to_monitors true 2024-05-15T13:04:22.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:22.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:22.542 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:04:22.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:04:22.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:22.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:04:23.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:24.152 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:24.152 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:24.152 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:04:24.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:04:24.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:24.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:04:25.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:26.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:26.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:26.011 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:04:26.012 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:04:26.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:26.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:04:26.499 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:27.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:27.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:27.501 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:04:27.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:04:27.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:27.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:04:28.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:29.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:29.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:29.059 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:04:29.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:04:29.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:29.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:04:29.588 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:30.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:30.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:30.590 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:04:30.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:04:30.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:30.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:04:31.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:32.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:32.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:32.058 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:04:32.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:04:32.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:32.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:04:32.683 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:33.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:33.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:33.686 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:04:33.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:04:33.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:33.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:04:34.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:35.118 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:35.118 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:35.118 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:04:35.119 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:04:35.119 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:35.119 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:04:35.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:36.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:36.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:36.672 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:04:36.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:04:36.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:36.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:04:37.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:38.162 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:38.162 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:38.163 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:04:38.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:04:38.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:38.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:04:39.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:40.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:40.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:40.059 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:04:40.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:04:40.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:40.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:04:40.835 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:41.836 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:41.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:41.837 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:04:41.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:04:41.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:41.838 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:04:43.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:44.017 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:44.017 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:44.017 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:04:44.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:04:44.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:44.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:04:44.913 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:45.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:45.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:45.916 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:04:45.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:04:45.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:45.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:04:47.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:48.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:48.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:48.075 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:04:48.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:04:48.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:48.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:04:48.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:04:49.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:04:49.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:49.852 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:04:49.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:04:49.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:49.853 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:04:50.328 INFO:tasks.workunit.client.0.smithi005.stdout:osd.5 up in weight 1 up_from 52 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3477847338,v1:127.0.0.1:6843/3477847338] [v2:127.0.0.1:6844/3477847338,v1:127.0.0.1:6845/3477847338] exists,up ae3b6776-1249-49a1-b629-d10018cefd97 2024-05-15T13:04:50.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:04:50.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:04:50.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:04:50.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:04:50.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-05-15T13:04:50.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:04:50.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:04:50.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-05-15T13:04:50.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:04:50.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-05-15T13:04:50.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-15T13:04:50.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:04:50.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:04:50.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:04:50.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-05-15T13:04:50.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-05-15T13:04:50.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:04:50.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:04:50.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:04:50.333 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:04:50.333 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:04:50.333 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:04:50.333 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:04:50.333 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:04:50.333 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:04:50.333 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:04:50.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:04:50.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:04:50.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:04:50.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:04:50.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:04:50.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:04:50.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:04:50.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:04:50.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:04:50.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:04:50.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:04:50.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:04:50.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-05-15T13:04:50.335 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:04:50.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=315fc867-7825-41d3-8d65-bdef4c2074c4 2024-05-15T13:04:50.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 315fc867-7825-41d3-8d65-bdef4c2074c4' 2024-05-15T13:04:50.335 INFO:tasks.workunit.client.0.smithi005.stdout:add osd6 315fc867-7825-41d3-8d65-bdef4c2074c4 2024-05-15T13:04:50.336 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:04:50.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDyskRm0KjwFRAA82iVvp+8wHVveWG+0V5GEg== 2024-05-15T13:04:50.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDyskRm0KjwFRAA82iVvp+8wHVveWG+0V5GEg=="}' 2024-05-15T13:04:50.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 315fc867-7825-41d3-8d65-bdef4c2074c4 -i td/test-erasure-eio/6/new.json 2024-05-15T13:04:51.217 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:04:51.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-05-15T13:04:51.238 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDyskRm0KjwFRAA82iVvp+8wHVveWG+0V5GEg== --osd-uuid 315fc867-7825-41d3-8d65-bdef4c2074c4 2024-05-15T13:04:51.276 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:51.285+0000 7eff112dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:04:51.364 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:51.373+0000 7eff112dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:04:51.365 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:51.373+0000 7eff112dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:04:51.366 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:51.373+0000 7eff112dd7c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-15T13:04:51.366 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:51.373+0000 7eff112dd7c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-15T13:04:51.366 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:51.373+0000 7eff112dd7c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-15T13:04:51.502 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:51.509+0000 7eff112dd7c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-05-15T13:04:58.996 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-05-15T13:04:58.996 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:04:58.997 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-05-15T13:04:58.997 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd6 key to auth repository 2024-05-15T13:04:58.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:04:59.572 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-05-15T13:04:59.572 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.6 2024-05-15T13:04:59.573 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:04:59.573 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:04:59.573 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:04:59.573 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:04:59.610 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:59.621+0000 7f813d18c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:04:59.662 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:59.669+0000 7f813d18c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:04:59.664 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:04:59.673+0000 7f813d18c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:04:59.992 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-05-15T13:04:59.992 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:04:59.992 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:04:59.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-05-15T13:04:59.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:04:59.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:04:59.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:04:59.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:04:59.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:04:59.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:05:00.229 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:05:00.237+0000 7f813d18c7c0 -1 Falling back to public interface 2024-05-15T13:05:00.430 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:05:01.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:05:01.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:05:01.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:05:01.432 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:05:01.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:05:01.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:05:01.857 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:05:02.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:05:02.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:05:02.859 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:05:02.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:05:02.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:05:02.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:05:02.893 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:05:02.901+0000 7f813d18c7c0 -1 osd.6 0 log_to_monitors true 2024-05-15T13:05:03.476 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:05:04.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:05:04.478 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:05:04.478 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:05:04.478 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:05:04.478 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:05:04.478 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:05:05.136 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:05:06.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:05:06.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:05:06.139 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:05:06.139 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:05:06.139 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:05:06.139 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:05:06.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:05:07.811 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:05:07.811 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:05:07.812 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:05:07.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:05:07.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:05:07.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:05:08.278 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:05:09.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:05:09.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:05:09.280 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:05:09.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:05:09.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:05:09.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:05:09.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:05:10.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:05:10.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:05:10.822 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:05:10.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:05:10.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:05:10.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:05:11.342 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:05:12.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:05:12.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:05:12.344 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:05:12.345 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:05:12.345 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:05:12.345 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:05:12.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:05:13.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:05:13.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:05:13.995 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:05:13.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:05:13.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:05:13.996 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:05:14.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:05:15.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:05:15.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:05:15.445 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:05:15.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:05:15.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:05:15.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:05:15.996 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:05:16.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:05:16.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:05:16.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:05:16.998 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:05:16.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:05:16.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:05:17.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:05:18.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:05:18.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:05:18.662 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:05:18.662 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:05:18.662 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:05:18.662 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:05:19.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:05:20.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:05:20.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:05:20.593 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:05:20.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:05:20.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:05:20.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:05:21.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:05:22.482 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:05:22.482 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:05:22.482 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:05:22.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:05:22.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:05:22.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:05:23.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:05:24.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:05:24.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:05:24.264 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:05:24.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:05:24.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:05:24.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:05:25.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:05:26.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:05:26.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:05:26.181 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:05:26.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:05:26.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:05:26.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:05:26.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:05:27.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:05:27.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:05:27.924 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:05:27.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:05:27.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:05:27.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:05:28.802 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:05:29.804 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:05:29.804 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:05:29.805 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:05:29.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:05:29.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:05:29.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:05:30.284 INFO:tasks.workunit.client.0.smithi005.stdout:osd.6 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3844466012,v1:127.0.0.1:6851/3844466012] [v2:127.0.0.1:6852/3844466012,v1:127.0.0.1:6853/3844466012] exists,up 315fc867-7825-41d3-8d65-bdef4c2074c4 2024-05-15T13:05:30.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:05:30.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:05:30.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:05:30.285 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-15T13:05:30.285 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T13:05:30.286 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T13:05:30.286 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:05:30.287 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:05:30.287 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:05:30.287 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.0.asok 2024-05-15T13:05:30.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-15T13:05:30.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-osd.0.asok log flush 2024-05-15T13:05:30.399 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T13:05:30.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-15T13:05:30.419 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T13:01:00.078+0000 7febe988e7c0 0 load: jerasure load: lrc 2024-05-15T13:05:30.420 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:457: TEST_ec_recovery_multiple_objects: CEPH_ARGS='--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:05:30.420 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:459: TEST_ec_recovery_multiple_objects: local poolname=pool-jerasure 2024-05-15T13:05:30.420 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:460: TEST_ec_recovery_multiple_objects: create_erasure_coded_pool pool-jerasure 3 2 2024-05-15T13:05:30.420 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-15T13:05:30.420 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-15T13:05:30.420 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-05-15T13:05:30.420 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-15T13:05:30.420 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-05-15T13:05:30.421 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-15T13:05:30.421 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-05-15T13:05:30.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-15T13:05:30.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-15T13:05:32.603 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-jerasure' already exists 2024-05-15T13:05:32.621 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T13:05:33.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-15T13:05:33.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:05:33.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:05:33.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:05:33.625 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:05:33.626 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:05:33.626 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:05:33.626 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:05:33.626 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:05:33.626 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:05:33.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:05:33.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:05:33.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:05:33.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:05:33.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:05:33.757 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:05:34.247 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:05:34.247 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:05:34.247 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:05:34.247 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T13:05:34.247 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T13:05:34.247 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T13:05:34.247 INFO:tasks.workunit.client.0.smithi005.stderr:6' 2024-05-15T13:05:34.248 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:05:34.248 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:05:34.248 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:05:34.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803829 2024-05-15T13:05:34.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803829 2024-05-15T13:05:34.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829' 2024-05-15T13:05:34.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:05:34.474 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:05:34.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574897 2024-05-15T13:05:34.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574897 2024-05-15T13:05:34.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-55834574897' 2024-05-15T13:05:34.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:05:34.639 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:05:34.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345960 2024-05-15T13:05:34.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345960 2024-05-15T13:05:34.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-55834574897 2-85899345960' 2024-05-15T13:05:34.951 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:05:34.951 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:05:35.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018914 2024-05-15T13:05:35.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018914 2024-05-15T13:05:35.174 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-55834574897 2-85899345960 3-128849018914' 2024-05-15T13:05:35.174 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:05:35.174 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T13:05:35.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659160 2024-05-15T13:05:35.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659160 2024-05-15T13:05:35.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-55834574897 2-85899345960 3-128849018914 4-176093659160' 2024-05-15T13:05:35.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:05:35.338 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-15T13:05:35.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=223338299410 2024-05-15T13:05:35.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 223338299410 2024-05-15T13:05:35.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-55834574897 2-85899345960 3-128849018914 4-176093659160 5-223338299410' 2024-05-15T13:05:35.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:05:35.492 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-15T13:05:35.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005066 2024-05-15T13:05:35.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005066 2024-05-15T13:05:35.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-55834574897 2-85899345960 3-128849018914 4-176093659160 5-223338299410 6-261993005066' 2024-05-15T13:05:35.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:05:35.655 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803829 2024-05-15T13:05:35.655 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:05:35.656 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:05:35.657 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803829 2024-05-15T13:05:35.657 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:05:35.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803829 2024-05-15T13:05:35.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803829' 2024-05-15T13:05:35.659 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 25769803829 2024-05-15T13:05:35.659 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:05:36.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803829 -lt 25769803829 2024-05-15T13:05:36.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:05:36.077 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574897 2024-05-15T13:05:36.077 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:05:36.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:05:36.079 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574897 2024-05-15T13:05:36.080 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:05:36.080 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574897 2024-05-15T13:05:36.081 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 55834574897 2024-05-15T13:05:36.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574897' 2024-05-15T13:05:36.081 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:05:36.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574897 -lt 55834574897 2024-05-15T13:05:36.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:05:36.493 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345960 2024-05-15T13:05:36.493 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:05:36.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:05:36.496 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345960 2024-05-15T13:05:36.496 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:05:36.497 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345960 2024-05-15T13:05:36.497 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 85899345960 2024-05-15T13:05:36.498 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345960' 2024-05-15T13:05:36.498 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:05:36.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345960 -lt 85899345960 2024-05-15T13:05:36.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:05:36.941 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-128849018914 2024-05-15T13:05:36.941 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:05:36.943 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:05:36.944 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-128849018914 2024-05-15T13:05:36.944 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:05:36.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018914 2024-05-15T13:05:36.945 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 128849018914 2024-05-15T13:05:36.946 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 128849018914' 2024-05-15T13:05:36.946 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:05:37.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018914 -lt 128849018914 2024-05-15T13:05:37.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:05:37.383 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-176093659160 2024-05-15T13:05:37.383 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:05:37.384 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T13:05:37.385 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-176093659160 2024-05-15T13:05:37.385 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:05:37.386 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.4 seq 176093659160 2024-05-15T13:05:37.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659160 2024-05-15T13:05:37.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 176093659160' 2024-05-15T13:05:37.387 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T13:05:37.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659160 -lt 176093659160 2024-05-15T13:05:37.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:05:37.822 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:05:37.823 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-223338299410 2024-05-15T13:05:37.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-15T13:05:37.824 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-223338299410 2024-05-15T13:05:37.824 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:05:37.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=223338299410 2024-05-15T13:05:37.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 223338299410' 2024-05-15T13:05:37.826 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.5 seq 223338299410 2024-05-15T13:05:37.826 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-15T13:05:38.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 223338299410 -lt 223338299410 2024-05-15T13:05:38.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:05:38.308 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-261993005066 2024-05-15T13:05:38.308 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:05:38.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-15T13:05:38.311 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-261993005066 2024-05-15T13:05:38.311 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:05:38.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005066 2024-05-15T13:05:38.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 261993005066' 2024-05-15T13:05:38.312 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.6 seq 261993005066 2024-05-15T13:05:38.313 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-15T13:05:38.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005066 -lt 261993005066 2024-05-15T13:05:38.749 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:05:38.749 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:05:38.749 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:05:39.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:05:39.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:05:39.316 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:05:39.316 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:05:39.316 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:05:39.316 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:05:39.316 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:05:39.317 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:05:39.715 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:05:39.715 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:05:39.715 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:05:39.715 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:05:40.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:05:40.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:05:40.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:05:40.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:462: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test1 2024-05-15T13:05:40.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-15T13:05:40.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-15T13:05:40.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2024-05-15T13:05:40.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:05:40.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-15T13:05:40.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:05:40.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-15T13:05:40.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:05:40.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-15T13:05:40.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:05:40.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-15T13:05:40.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2024-05-15T13:05:40.470 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:463: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test2 2024-05-15T13:05:40.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-15T13:05:40.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-15T13:05:40.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2024-05-15T13:05:40.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:05:40.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-15T13:05:40.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:05:40.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-15T13:05:40.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:05:40.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-15T13:05:40.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:05:40.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-15T13:05:40.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2024-05-15T13:05:40.788 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:464: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test3 2024-05-15T13:05:40.789 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-15T13:05:40.789 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-15T13:05:40.789 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2024-05-15T13:05:40.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:05:40.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-15T13:05:40.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:05:40.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-15T13:05:40.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:05:40.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-15T13:05:40.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:05:40.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-15T13:05:40.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2024-05-15T13:05:41.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:466: TEST_ec_recovery_multiple_objects: ceph osd out 0 2024-05-15T13:05:41.789 INFO:tasks.workunit.client.0.smithi005.stderr:osd.0 is already out. 2024-05-15T13:05:41.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:469: TEST_ec_recovery_multiple_objects: wait_for_clean 2024-05-15T13:05:41.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:05:41.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:05:41.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:05:41.810 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:05:41.810 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:05:41.810 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:05:41.811 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:05:41.811 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:05:41.811 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:05:41.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:05:41.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:05:41.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:05:41.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:05:41.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:05:41.927 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:05:42.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:05:42.767 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:05:42.768 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:05:42.768 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T13:05:42.768 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T13:05:42.768 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T13:05:42.768 INFO:tasks.workunit.client.0.smithi005.stderr:6' 2024-05-15T13:05:42.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:05:42.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:05:42.768 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:05:43.064 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803832 2024-05-15T13:05:43.064 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803832 2024-05-15T13:05:43.064 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803832' 2024-05-15T13:05:43.064 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:05:43.064 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:05:43.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574901 2024-05-15T13:05:43.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574901 2024-05-15T13:05:43.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803832 1-55834574901' 2024-05-15T13:05:43.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:05:43.411 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:05:43.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345964 2024-05-15T13:05:43.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345964 2024-05-15T13:05:43.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803832 1-55834574901 2-85899345964' 2024-05-15T13:05:43.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:05:43.571 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:05:43.789 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018918 2024-05-15T13:05:43.789 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018918 2024-05-15T13:05:43.789 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803832 1-55834574901 2-85899345964 3-128849018918' 2024-05-15T13:05:43.789 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:05:43.790 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T13:05:43.947 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659163 2024-05-15T13:05:43.947 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659163 2024-05-15T13:05:43.947 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803832 1-55834574901 2-85899345964 3-128849018918 4-176093659163' 2024-05-15T13:05:43.947 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:05:43.947 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-15T13:05:44.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=223338299413 2024-05-15T13:05:44.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 223338299413 2024-05-15T13:05:44.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803832 1-55834574901 2-85899345964 3-128849018918 4-176093659163 5-223338299413' 2024-05-15T13:05:44.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:05:44.102 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-15T13:05:44.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005069 2024-05-15T13:05:44.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005069 2024-05-15T13:05:44.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803832 1-55834574901 2-85899345964 3-128849018918 4-176093659163 5-223338299413 6-261993005069' 2024-05-15T13:05:44.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:05:44.258 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803832 2024-05-15T13:05:44.258 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:05:44.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:05:44.259 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803832 2024-05-15T13:05:44.260 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:05:44.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803832 2024-05-15T13:05:44.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803832' 2024-05-15T13:05:44.261 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 25769803832 2024-05-15T13:05:44.261 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:05:44.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803832 -lt 25769803832 2024-05-15T13:05:44.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:05:44.676 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574901 2024-05-15T13:05:44.677 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:05:44.679 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:05:44.679 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574901 2024-05-15T13:05:44.680 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:05:44.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574901 2024-05-15T13:05:44.681 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 55834574901 2024-05-15T13:05:44.682 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574901' 2024-05-15T13:05:44.682 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:05:45.103 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574899 -lt 55834574901 2024-05-15T13:05:45.103 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-15T13:05:46.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-15T13:05:46.105 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:05:46.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574901 -lt 55834574901 2024-05-15T13:05:46.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:05:46.538 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345964 2024-05-15T13:05:46.538 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:05:46.539 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:05:46.540 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345964 2024-05-15T13:05:46.540 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:05:46.542 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 85899345964 2024-05-15T13:05:46.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345964 2024-05-15T13:05:46.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345964' 2024-05-15T13:05:46.542 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:05:46.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345964 -lt 85899345964 2024-05-15T13:05:46.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:05:46.999 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-128849018918 2024-05-15T13:05:46.999 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:05:47.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:05:47.002 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-128849018918 2024-05-15T13:05:47.002 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:05:47.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018918 2024-05-15T13:05:47.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 128849018918' 2024-05-15T13:05:47.004 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 128849018918 2024-05-15T13:05:47.004 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:05:47.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018918 -lt 128849018918 2024-05-15T13:05:47.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:05:47.450 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-176093659163 2024-05-15T13:05:47.450 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:05:47.452 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T13:05:47.453 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-176093659163 2024-05-15T13:05:47.453 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:05:47.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659163 2024-05-15T13:05:47.454 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.4 seq 176093659163 2024-05-15T13:05:47.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 176093659163' 2024-05-15T13:05:47.455 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T13:05:47.903 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659164 -lt 176093659163 2024-05-15T13:05:47.903 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:05:47.904 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-223338299413 2024-05-15T13:05:47.904 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:05:47.905 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-15T13:05:47.906 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-223338299413 2024-05-15T13:05:47.906 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:05:47.907 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=223338299413 2024-05-15T13:05:47.907 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.5 seq 223338299413 2024-05-15T13:05:47.908 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 223338299413' 2024-05-15T13:05:47.908 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-15T13:05:48.351 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 223338299414 -lt 223338299413 2024-05-15T13:05:48.351 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:05:48.352 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-261993005069 2024-05-15T13:05:48.352 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:05:48.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-15T13:05:48.355 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-261993005069 2024-05-15T13:05:48.355 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:05:48.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005069 2024-05-15T13:05:48.356 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.6 seq 261993005069 2024-05-15T13:05:48.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 261993005069' 2024-05-15T13:05:48.357 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-15T13:05:48.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005070 -lt 261993005069 2024-05-15T13:05:48.812 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:05:48.812 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:05:48.812 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:05:49.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:05:49.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:05:49.404 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:05:49.404 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:05:49.404 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:05:49.404 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:05:49.404 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:05:49.404 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:05:49.857 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:05:49.857 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:05:49.858 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:05:49.858 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:05:50.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:05:50.404 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:05:50.404 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:05:50.404 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:471: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test1 2024-05-15T13:05:50.404 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-15T13:05:50.404 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-15T13:05:50.404 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2024-05-15T13:05:50.404 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-15T13:05:50.404 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-15T13:05:50.404 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2024-05-15T13:05:50.512 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-15T13:05:50.514 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-15T13:05:50.516 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:472: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test2 2024-05-15T13:05:50.516 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-15T13:05:50.516 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-15T13:05:50.516 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2024-05-15T13:05:50.516 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-15T13:05:50.516 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-15T13:05:50.516 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2024-05-15T13:05:50.572 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-15T13:05:50.573 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-15T13:05:50.574 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:473: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test3 2024-05-15T13:05:50.574 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-15T13:05:50.574 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-15T13:05:50.574 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2024-05-15T13:05:50.574 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-15T13:05:50.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-15T13:05:50.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2024-05-15T13:05:50.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-15T13:05:50.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-15T13:05:50.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:475: TEST_ec_recovery_multiple_objects: delete_erasure_coded_pool pool-jerasure 2024-05-15T13:05:50.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-15T13:05:50.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-15T13:05:51.097 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-jerasure' does not exist 2024-05-15T13:05:51.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-15T13:05:51.607 INFO:tasks.workunit.client.0.smithi005.stderr:erasure-code-profile myprofile does not exist 2024-05-15T13:05:51.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-15T13:05:51.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-15T13:05:51.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T13:05:51.625 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-15T13:05:51.625 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:05:51.625 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:05:51.626 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:05:51.626 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:05:51.626 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:05:51.936 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:05:51.937 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T13:05:51.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T13:05:51.939 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T13:05:51.941 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T13:05:51.942 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T13:05:51.942 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T13:05:51.943 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:05:51.943 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T13:05:51.943 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T13:05:51.944 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:05:51.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:05:51.946 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T13:05:51.946 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-15T13:05:51.983 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T13:05:51.984 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:05:51.984 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:05:51.984 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80702 2024-05-15T13:05:51.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T13:05:51.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T13:05:51.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-15T13:05:51.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-15T13:05:51.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-15T13:05:51.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-15T13:05:51.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-15T13:05:51.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T13:05:51.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-15T13:05:51.986 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:05:51.986 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:05:51.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:05:51.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:05:51.987 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:05:51.988 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:05:51.989 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T13:05:51.989 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T13:05:51.990 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T13:05:51.991 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T13:05:51.991 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T13:05:51.991 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T13:05:51.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:05:51.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T13:05:51.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T13:05:51.993 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:05:51.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:05:51.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T13:05:51.996 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-15T13:05:51.997 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T13:05:51.997 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:05:51.997 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:05:51.997 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80702 2024-05-15T13:05:51.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T13:05:51.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T13:05:51.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-15T13:05:51.999 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T13:05:52.000 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:05:52.000 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:05:52.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.80702 2024-05-15T13:05:52.003 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T13:05:52.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-15T13:05:52.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T13:05:52.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-15T13:05:52.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-15T13:05:52.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-15T13:05:52.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T13:05:52.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T13:05:52.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T13:05:52.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-15T13:05:52.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-15T13:05:52.426 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T13:05:52.426 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:05:52.426 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:05:52.427 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:05:52.427 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:05:52.427 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:05:52.427 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:05:52.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-15T13:05:52.621 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T13:05:52.622 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T13:05:52.622 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T13:05:52.622 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T13:05:52.622 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T13:05:52.623 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T13:05:52.623 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T13:05:52.623 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T13:05:52.623 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T13:05:52.623 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:05:52.624 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:05:52.624 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:05:52.624 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T13:05:52.624 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T13:05:52.624 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.80702/ceph-mon.a.asok config get fsid 2024-05-15T13:05:52.742 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T13:05:52.743 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T13:05:52.743 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T13:05:52.743 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T13:05:52.743 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T13:05:52.744 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T13:05:52.744 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T13:05:52.744 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T13:05:52.744 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:05:52.744 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:05:52.744 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:05:52.744 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T13:05:52.745 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T13:05:52.745 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.80702/ceph-mon.a.asok config get mon_host 2024-05-15T13:05:52.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-15T13:05:52.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-15T13:05:52.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T13:05:52.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T13:05:52.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T13:05:52.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-15T13:05:52.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T13:05:53.779 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T13:05:53.779 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:05:53.779 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:05:53.779 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:05:53.779 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:05:53.779 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:05:53.780 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:05:53.780 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T13:05:53.781 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T13:05:53.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-15T13:05:53.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-15T13:05:54.191 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd' already exists 2024-05-15T13:05:54.207 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T13:05:55.210 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-15T13:05:55.210 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T13:05:55.210 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T13:05:55.211 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:05:55.211 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:05:55.211 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:05:55.211 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T13:05:55.211 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-15T13:05:55.212 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-mon.a.asok log flush 2024-05-15T13:05:55.314 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T13:05:55.326 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-15T13:05:55.328 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T13:05:52.484+0000 7f8b62be4c80 0 load: jerasure load: lrc 2024-05-15T13:05:55.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects_eio td/test-erasure-eio 2024-05-15T13:05:55.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:480: TEST_ec_recovery_multiple_objects_eio: local dir=td/test-erasure-eio 2024-05-15T13:05:55.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:481: TEST_ec_recovery_multiple_objects_eio: local objname=myobject 2024-05-15T13:05:55.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:483: TEST_ec_recovery_multiple_objects_eio: ORIG_ARGS='--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:05:55.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:484: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-15T13:05:55.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:485: TEST_ec_recovery_multiple_objects_eio: setup_osds 7 2024-05-15T13:05:55.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-05-15T13:05:55.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-15T13:05:55.330 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-05-15T13:05:55.331 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-05-15T13:05:55.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:05:55.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-15T13:05:55.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:05:55.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:05:55.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T13:05:55.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:05:55.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-15T13:05:55.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-15T13:05:55.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:05:55.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:05:55.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:05:55.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-15T13:05:55.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-15T13:05:55.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:05:55.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:05:55.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:05:55.334 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:05:55.335 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:05:55.335 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:05:55.336 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:05:55.336 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:05:55.336 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:05:55.337 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:05:55.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:05:55.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:05:55.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:05:55.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:05:55.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:05:55.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:05:55.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:05:55.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:05:55.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:05:55.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:05:55.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:05:55.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:05:55.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-15T13:05:55.337 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:05:55.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=09e0a3b5-ad53-4b61-b0f0-d0525086d4a1 2024-05-15T13:05:55.338 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 09e0a3b5-ad53-4b61-b0f0-d0525086d4a1 2024-05-15T13:05:55.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 09e0a3b5-ad53-4b61-b0f0-d0525086d4a1' 2024-05-15T13:05:55.339 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:05:55.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAzs0Rm98mVFhAAoMwgX4w1Hi/bKrdUHA7daA== 2024-05-15T13:05:55.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAzs0Rm98mVFhAAoMwgX4w1Hi/bKrdUHA7daA=="}' 2024-05-15T13:05:55.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 09e0a3b5-ad53-4b61-b0f0-d0525086d4a1 -i td/test-erasure-eio/0/new.json 2024-05-15T13:05:55.717 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:05:55.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-15T13:05:55.736 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAzs0Rm98mVFhAAoMwgX4w1Hi/bKrdUHA7daA== --osd-uuid 09e0a3b5-ad53-4b61-b0f0-d0525086d4a1 2024-05-15T13:05:55.783 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:05:55.796+0000 7f987f30d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:05:55.794 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:05:55.804+0000 7f987f30d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:05:55.795 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:05:55.808+0000 7f987f30d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:05:55.796 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:05:55.808+0000 7f987f30d7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T13:05:55.796 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:05:55.808+0000 7f987f30d7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T13:05:55.796 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:05:55.808+0000 7f987f30d7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T13:05:55.834 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:05:55.844+0000 7f987f30d7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-15T13:06:02.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-15T13:06:02.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:06:02.547 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T13:06:02.547 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T13:06:02.547 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:06:03.115 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T13:06:03.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T13:06:03.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:06:03.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:06:03.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:06:03.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:06:03.153 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:06:03.163+0000 7f5d319057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:06:03.155 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:06:03.167+0000 7f5d319057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:06:03.156 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:06:03.167+0000 7f5d319057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:06:03.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T13:06:03.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:06:03.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T13:06:03.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:06:03.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:06:03.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:03.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:06:03.602 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:06:03.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:03.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:06:03.740 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:06:03.751+0000 7f5d319057c0 -1 Falling back to public interface 2024-05-15T13:06:04.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:05.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:05.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:05.044 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:06:05.044 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:06:05.044 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:05.044 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:06:05.475 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:06.413 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:06:06.423+0000 7f5d319057c0 -1 osd.0 0 log_to_monitors true 2024-05-15T13:06:06.476 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:06.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:06.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:06:06.477 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:06:06.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:06.478 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:06:07.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:08.177 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:08.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:08.178 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:06:08.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:06:08.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:08.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:06:08.621 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:09.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:09.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:09.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:06:09.623 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:06:09.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:09.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:06:10.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:11.097 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:11.097 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:11.097 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:06:11.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:06:11.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:11.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:06:11.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:12.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:12.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:12.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:06:12.633 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:06:12.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:12.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:06:13.078 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:14.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:14.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:14.083 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:06:14.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:06:14.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:14.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:06:14.620 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:15.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:15.622 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:06:15.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:15.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:06:15.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:15.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:06:16.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:17.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:17.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:17.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:06:17.085 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:06:17.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:17.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:06:17.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:18.687 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:18.687 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:18.687 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:06:18.688 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:06:18.688 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:18.688 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:06:19.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:20.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:20.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:20.256 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:06:20.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:06:20.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:20.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:06:21.069 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:22.070 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:22.071 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:22.071 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:06:22.071 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:06:22.071 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:22.072 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:06:23.024 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:24.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:24.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:24.026 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:06:24.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:06:24.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:24.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:06:24.846 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:25.848 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:25.848 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:25.848 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:06:25.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:06:25.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:25.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:06:26.782 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:27.784 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:27.784 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:27.784 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:06:27.784 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:06:27.784 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:27.785 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:06:28.898 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:29.900 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:29.900 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:29.901 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:06:29.901 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:06:29.901 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:29.901 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:06:31.019 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:32.021 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:32.021 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:32.021 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:06:32.022 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:06:32.022 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:32.022 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:06:32.628 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3915007686,v1:127.0.0.1:6803/3915007686] [v2:127.0.0.1:6804/3915007686,v1:127.0.0.1:6805/3915007686] exists,up 09e0a3b5-ad53-4b61-b0f0-d0525086d4a1 2024-05-15T13:06:32.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:06:32.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:06:32.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:06:32.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:06:32.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-15T13:06:32.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:06:32.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:06:32.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-15T13:06:32.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:06:32.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-15T13:06:32.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-15T13:06:32.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:06:32.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:06:32.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:06:32.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-15T13:06:32.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-15T13:06:32.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:06:32.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:06:32.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:06:32.632 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:06:32.632 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:06:32.632 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:06:32.632 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:06:32.632 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:06:32.632 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:06:32.632 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:06:32.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:06:32.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:06:32.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:06:32.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:06:32.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:06:32.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:06:32.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:06:32.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:06:32.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:06:32.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:06:32.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:06:32.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:06:32.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-15T13:06:32.634 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:06:32.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=108ae43a-606e-4267-a404-7c44316d0560 2024-05-15T13:06:32.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 108ae43a-606e-4267-a404-7c44316d0560' 2024-05-15T13:06:32.635 INFO:tasks.workunit.client.0.smithi005.stdout:add osd1 108ae43a-606e-4267-a404-7c44316d0560 2024-05-15T13:06:32.635 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:06:32.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBYs0RmMTIVKBAA5STBDSi+ck+gleakKKs+4g== 2024-05-15T13:06:32.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBYs0RmMTIVKBAA5STBDSi+ck+gleakKKs+4g=="}' 2024-05-15T13:06:32.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 108ae43a-606e-4267-a404-7c44316d0560 -i td/test-erasure-eio/1/new.json 2024-05-15T13:06:33.345 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:06:33.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-15T13:06:33.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBYs0RmMTIVKBAA5STBDSi+ck+gleakKKs+4g== --osd-uuid 108ae43a-606e-4267-a404-7c44316d0560 2024-05-15T13:06:33.398 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:06:33.407+0000 7fe83caf67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:06:33.405 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:06:33.415+0000 7fe83caf67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:06:33.406 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:06:33.415+0000 7fe83caf67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:06:33.407 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:06:33.419+0000 7fe83caf67c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T13:06:33.407 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:06:33.419+0000 7fe83caf67c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T13:06:33.407 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:06:33.419+0000 7fe83caf67c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T13:06:33.436 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:06:33.447+0000 7fe83caf67c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-15T13:06:40.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-15T13:06:40.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:06:40.281 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd1 key to auth repository 2024-05-15T13:06:40.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-15T13:06:40.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:06:40.887 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.1 2024-05-15T13:06:40.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-15T13:06:40.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:06:40.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:06:40.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:06:40.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:06:40.924 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:06:40.935+0000 7fd308a877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:06:40.960 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:06:40.971+0000 7fd308a877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:06:40.962 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:06:40.971+0000 7fd308a877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:06:41.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-15T13:06:41.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:06:41.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-15T13:06:41.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:06:41.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:06:41.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:41.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:06:41.332 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:06:41.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:06:41.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:41.528 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:06:41.539+0000 7fd308a877c0 -1 Falling back to public interface 2024-05-15T13:06:41.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:42.759 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:42.759 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:42.759 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:06:42.759 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:06:42.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:42.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:06:43.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:44.189 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:06:44.199+0000 7fd308a877c0 -1 osd.1 0 log_to_monitors true 2024-05-15T13:06:44.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:44.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:44.235 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:06:44.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:06:44.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:06:44.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:45.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:46.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:46.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:46.057 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:06:46.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:06:46.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:46.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:06:46.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:47.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:47.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:47.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:06:47.570 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:06:47.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:47.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:06:48.131 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:49.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:49.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:49.133 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:06:49.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:06:49.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:49.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:06:49.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:50.637 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:50.637 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:50.637 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:06:50.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:06:50.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:50.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:06:51.197 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:52.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:52.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:52.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:06:52.199 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:06:52.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:52.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:06:52.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:53.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:53.674 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:53.674 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:06:53.674 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:06:53.674 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:53.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:06:54.213 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:55.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:55.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:55.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:06:55.215 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:06:55.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:55.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:06:55.693 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:56.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:56.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:56.695 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:06:56.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:06:56.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:56.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:06:57.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:58.196 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:58.196 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:58.197 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:06:58.197 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:06:58.197 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:58.197 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:06:58.732 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:06:59.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:06:59.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:06:59.734 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:06:59.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:06:59.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:06:59.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:07:00.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:01.406 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:01.406 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:01.407 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:07:01.407 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:07:01.407 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:01.407 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:07:02.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:03.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:03.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:03.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:07:03.405 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:07:03.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:03.406 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:07:04.143 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:05.145 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:05.145 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:05.145 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:07:05.146 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:07:05.146 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:05.146 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:07:05.973 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:06.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:06.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:06.976 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:07:06.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:07:06.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:06.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:07:07.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:08.404 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:08.404 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:08.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:07:08.405 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:07:08.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:08.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:07:09.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:10.017 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:10.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:10.018 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:07:10.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:07:10.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:10.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:07:10.532 INFO:tasks.workunit.client.0.smithi005.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2935631312,v1:127.0.0.1:6811/2935631312] [v2:127.0.0.1:6812/2935631312,v1:127.0.0.1:6813/2935631312] exists,up 108ae43a-606e-4267-a404-7c44316d0560 2024-05-15T13:07:10.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:07:10.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:07:10.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:07:10.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:07:10.534 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-15T13:07:10.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:07:10.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:07:10.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-15T13:07:10.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:07:10.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-15T13:07:10.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-15T13:07:10.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:07:10.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:07:10.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:07:10.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-15T13:07:10.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-15T13:07:10.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:07:10.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:07:10.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:07:10.536 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:07:10.536 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:07:10.536 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:07:10.536 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:07:10.536 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:07:10.536 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:07:10.537 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:07:10.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:07:10.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:07:10.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:07:10.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:07:10.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:07:10.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:07:10.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:07:10.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:07:10.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:07:10.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:07:10.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:07:10.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:07:10.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-15T13:07:10.538 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:07:10.539 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=054df59e-fc60-478d-acfe-f170b802faa5 2024-05-15T13:07:10.539 INFO:tasks.workunit.client.0.smithi005.stdout:add osd2 054df59e-fc60-478d-acfe-f170b802faa5 2024-05-15T13:07:10.539 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 054df59e-fc60-478d-acfe-f170b802faa5' 2024-05-15T13:07:10.539 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:07:10.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB+s0RmzZFaIhAA3oz5AYUSuetKaFdIXUeoJg== 2024-05-15T13:07:10.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB+s0RmzZFaIhAA3oz5AYUSuetKaFdIXUeoJg=="}' 2024-05-15T13:07:10.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 054df59e-fc60-478d-acfe-f170b802faa5 -i td/test-erasure-eio/2/new.json 2024-05-15T13:07:11.262 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:07:11.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-15T13:07:11.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB+s0RmzZFaIhAA3oz5AYUSuetKaFdIXUeoJg== --osd-uuid 054df59e-fc60-478d-acfe-f170b802faa5 2024-05-15T13:07:11.322 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:11.330+0000 7f4b5a9b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:07:11.356 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:11.366+0000 7f4b5a9b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:07:11.357 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:11.366+0000 7f4b5a9b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:07:11.358 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:11.366+0000 7f4b5a9b67c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T13:07:11.358 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:11.366+0000 7f4b5a9b67c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T13:07:11.358 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:11.366+0000 7f4b5a9b67c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T13:07:11.402 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:11.410+0000 7f4b5a9b67c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-15T13:07:18.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-15T13:07:18.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:07:18.482 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-15T13:07:18.482 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd2 key to auth repository 2024-05-15T13:07:18.482 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:07:19.049 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.2 2024-05-15T13:07:19.049 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-15T13:07:19.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:07:19.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:07:19.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:07:19.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:07:19.087 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:19.098+0000 7f45b0e437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:07:19.127 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:19.138+0000 7f45b0e437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:07:19.129 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:19.138+0000 7f45b0e437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:07:19.500 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-15T13:07:19.500 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:07:19.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-15T13:07:19.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:07:19.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:07:19.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:19.501 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:07:19.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:07:19.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:07:19.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:19.696 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:19.706+0000 7f45b0e437c0 -1 Falling back to public interface 2024-05-15T13:07:19.941 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:20.942 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:20.943 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:20.943 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:07:20.943 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:07:20.943 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:20.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:07:21.376 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:22.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:22.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:22.378 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:07:22.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:07:22.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:22.379 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:07:22.385 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:22.394+0000 7f45b0e437c0 -1 osd.2 0 log_to_monitors true 2024-05-15T13:07:22.973 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:23.974 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:23.974 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:23.974 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:07:23.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:07:23.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:23.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:07:24.453 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:25.455 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:25.455 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:25.456 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:07:25.456 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:07:25.456 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:25.456 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:07:25.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:26.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:26.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:26.924 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:07:26.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:07:26.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:26.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:07:27.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:28.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:28.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:28.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:07:28.364 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:07:28.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:28.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:07:28.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:29.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:29.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:29.823 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:07:29.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:07:29.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:29.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:07:30.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:31.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:31.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:31.319 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:07:31.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:07:31.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:31.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:07:31.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:32.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:32.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:32.882 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:07:32.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:07:32.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:32.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:07:33.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:34.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:34.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:34.441 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:07:34.441 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:07:34.441 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:34.441 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:07:34.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:35.941 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:35.942 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:35.942 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:07:35.942 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:07:35.942 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:35.943 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:07:36.806 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:37.808 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:37.808 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:37.808 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:07:37.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:07:37.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:37.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:07:38.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:39.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:39.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:39.705 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:07:39.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:07:39.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:39.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:07:40.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:41.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:41.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:41.695 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:07:41.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:07:41.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:41.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:07:42.386 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:43.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:43.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:43.389 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:07:43.389 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:07:43.389 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:43.389 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:07:44.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:45.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:45.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:45.259 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:07:45.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:07:45.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:45.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:07:46.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:47.219 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:47.219 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:47.219 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:07:47.220 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:07:47.220 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:47.220 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:07:47.848 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:48.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:48.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:48.850 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:07:48.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:07:48.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:48.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:07:49.422 INFO:tasks.workunit.client.0.smithi005.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1067979120,v1:127.0.0.1:6819/1067979120] [v2:127.0.0.1:6820/1067979120,v1:127.0.0.1:6821/1067979120] exists,up 054df59e-fc60-478d-acfe-f170b802faa5 2024-05-15T13:07:49.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:07:49.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:07:49.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:07:49.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:07:49.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-15T13:07:49.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:07:49.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:07:49.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-15T13:07:49.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:07:49.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-15T13:07:49.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-15T13:07:49.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:07:49.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:07:49.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:07:49.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-15T13:07:49.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-15T13:07:49.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:07:49.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:07:49.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:07:49.428 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:07:49.428 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:07:49.428 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:07:49.428 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:07:49.428 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:07:49.428 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:07:49.428 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:07:49.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:07:49.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:07:49.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:07:49.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:07:49.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:07:49.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:07:49.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:07:49.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:07:49.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:07:49.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:07:49.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:07:49.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:07:49.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-15T13:07:49.430 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:07:49.430 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=782bdb50-7177-480d-b57c-a7a2972b3af2 2024-05-15T13:07:49.430 INFO:tasks.workunit.client.0.smithi005.stdout:add osd3 782bdb50-7177-480d-b57c-a7a2972b3af2 2024-05-15T13:07:49.430 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 782bdb50-7177-480d-b57c-a7a2972b3af2' 2024-05-15T13:07:49.431 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:07:49.457 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCls0RmGPHtGxAAi+ItnVb59vpe04Qg5T9ljg== 2024-05-15T13:07:49.457 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCls0RmGPHtGxAAi+ItnVb59vpe04Qg5T9ljg=="}' 2024-05-15T13:07:49.457 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 782bdb50-7177-480d-b57c-a7a2972b3af2 -i td/test-erasure-eio/3/new.json 2024-05-15T13:07:50.170 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:07:50.187 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-15T13:07:50.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCls0RmGPHtGxAAi+ItnVb59vpe04Qg5T9ljg== --osd-uuid 782bdb50-7177-480d-b57c-a7a2972b3af2 2024-05-15T13:07:50.224 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:50.233+0000 7f7fa439f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:07:50.340 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:50.349+0000 7f7fa439f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:07:50.342 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:50.349+0000 7f7fa439f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:07:50.342 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:50.349+0000 7f7fa439f7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T13:07:50.342 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:50.349+0000 7f7fa439f7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T13:07:50.342 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:50.349+0000 7f7fa439f7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T13:07:50.461 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:50.469+0000 7f7fa439f7c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-15T13:07:56.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-15T13:07:56.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:07:56.466 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd3 key to auth repository 2024-05-15T13:07:56.466 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-15T13:07:56.466 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:07:57.039 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.3 2024-05-15T13:07:57.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-15T13:07:57.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:07:57.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:07:57.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:07:57.041 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:07:57.077 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:57.085+0000 7fda90b757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:07:57.080 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:57.089+0000 7fda90b757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:07:57.081 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:57.089+0000 7fda90b757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:07:57.525 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-15T13:07:57.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:07:57.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-15T13:07:57.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:07:57.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:07:57.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:57.526 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:07:57.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:07:57.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:07:57.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:57.630 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:07:57.637+0000 7fda90b757c0 -1 Falling back to public interface 2024-05-15T13:07:57.968 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:07:58.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:07:58.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:07:58.970 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:07:58.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:07:58.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:07:58.971 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:07:59.395 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:00.178 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:08:00.185+0000 7fda90b757c0 -1 osd.3 0 log_to_monitors true 2024-05-15T13:08:00.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:00.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:00.398 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:08:00.398 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:08:00.398 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:00.398 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:08:01.069 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:02.071 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:02.071 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:02.072 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:08:02.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:08:02.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:02.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:08:02.736 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:03.737 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:03.737 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:03.738 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:08:03.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:08:03.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:03.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:08:04.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:05.245 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:05.245 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:05.245 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:08:05.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:08:05.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:05.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:08:05.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:06.737 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:06.737 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:06.737 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:08:06.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:08:06.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:06.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:08:07.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:08.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:08.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:08.312 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:08:08.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:08:08.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:08.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:08:08.755 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:09.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:09.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:09.757 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:08:09.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:08:09.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:09.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:08:10.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:11.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:11.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:11.264 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:08:11.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:08:11.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:11.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:08:11.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:12.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:12.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:12.764 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:08:12.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:08:12.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:12.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:08:13.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:14.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:14.253 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:14.253 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:08:14.253 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:08:14.253 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:14.253 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:08:14.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:15.897 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:15.897 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:15.897 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:08:15.898 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:08:15.898 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:15.898 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:08:17.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:18.146 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:18.146 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:18.147 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:08:18.147 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:08:18.147 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:18.147 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:08:18.992 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:19.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:19.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:19.994 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:08:19.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:08:19.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:19.995 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:08:20.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:21.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:21.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:21.880 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:08:21.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:08:21.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:21.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:08:22.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:23.732 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:23.732 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:23.732 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:08:23.732 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:08:23.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:23.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:08:24.626 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:25.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:25.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:25.629 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:08:25.629 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:08:25.629 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:25.629 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:08:26.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:27.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:27.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:27.085 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:08:27.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:08:27.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:27.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:08:27.601 INFO:tasks.workunit.client.0.smithi005.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1404240685,v1:127.0.0.1:6827/1404240685] [v2:127.0.0.1:6828/1404240685,v1:127.0.0.1:6829/1404240685] exists,up 782bdb50-7177-480d-b57c-a7a2972b3af2 2024-05-15T13:08:27.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:08:27.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:08:27.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:08:27.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:08:27.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-05-15T13:08:27.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:08:27.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:08:27.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-05-15T13:08:27.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:08:27.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-05-15T13:08:27.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-15T13:08:27.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:08:27.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:08:27.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:08:27.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-05-15T13:08:27.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-05-15T13:08:27.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:08:27.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:08:27.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:08:27.605 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:08:27.605 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:08:27.605 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:08:27.605 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:08:27.605 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:08:27.606 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:08:27.606 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:08:27.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:08:27.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:08:27.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:08:27.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:08:27.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:08:27.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:08:27.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:08:27.608 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:08:27.608 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:08:27.608 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:08:27.608 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:08:27.608 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:08:27.608 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-05-15T13:08:27.608 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:08:27.608 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2760e0d4-4a20-4d2c-86be-e3faa5108f12 2024-05-15T13:08:27.609 INFO:tasks.workunit.client.0.smithi005.stdout:add osd4 2760e0d4-4a20-4d2c-86be-e3faa5108f12 2024-05-15T13:08:27.609 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 2760e0d4-4a20-4d2c-86be-e3faa5108f12' 2024-05-15T13:08:27.609 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:08:27.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDLs0RmYQNQJhAA/ChYU0mMvMos6kPW099J+g== 2024-05-15T13:08:27.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDLs0RmYQNQJhAA/ChYU0mMvMos6kPW099J+g=="}' 2024-05-15T13:08:27.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2760e0d4-4a20-4d2c-86be-e3faa5108f12 -i td/test-erasure-eio/4/new.json 2024-05-15T13:08:28.737 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:08:28.759 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-05-15T13:08:28.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDLs0RmYQNQJhAA/ChYU0mMvMos6kPW099J+g== --osd-uuid 2760e0d4-4a20-4d2c-86be-e3faa5108f12 2024-05-15T13:08:28.797 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:08:28.804+0000 7f54884c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:08:28.819 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:08:28.824+0000 7f54884c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:08:28.821 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:08:28.828+0000 7f54884c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:08:28.821 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:08:28.828+0000 7f54884c37c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T13:08:28.821 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:08:28.828+0000 7f54884c37c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T13:08:28.821 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:08:28.828+0000 7f54884c37c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T13:08:28.999 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:08:29.004+0000 7f54884c37c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-05-15T13:08:36.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-05-15T13:08:36.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:08:36.614 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd4 key to auth repository 2024-05-15T13:08:36.615 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-05-15T13:08:36.615 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:08:37.192 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.4 2024-05-15T13:08:37.192 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-05-15T13:08:37.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:08:37.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:08:37.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:08:37.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:08:37.231 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:08:37.236+0000 7f32e9fbf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:08:37.234 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:08:37.240+0000 7f32e9fbf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:08:37.235 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:08:37.244+0000 7f32e9fbf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:08:37.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-05-15T13:08:37.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:08:37.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-05-15T13:08:37.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:08:37.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:08:37.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:37.636 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:08:37.637 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:08:37.637 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:37.637 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:08:37.801 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:08:37.808+0000 7f32e9fbf7c0 -1 Falling back to public interface 2024-05-15T13:08:38.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:39.078 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:39.078 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:39.078 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:08:39.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:08:39.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:39.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:08:39.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:40.490 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:08:40.496+0000 7f32e9fbf7c0 -1 osd.4 0 log_to_monitors true 2024-05-15T13:08:40.498 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:40.498 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:40.498 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:08:40.498 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:08:40.498 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:40.498 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:08:41.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:42.078 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:42.078 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:42.079 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:08:42.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:08:42.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:42.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:08:42.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:43.972 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:43.972 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:43.972 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:08:43.973 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:08:43.973 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:43.973 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:08:44.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:45.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:45.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:45.550 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:08:45.551 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:08:45.551 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:45.551 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:08:46.162 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:47.164 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:47.164 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:47.164 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:08:47.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:08:47.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:47.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:08:47.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:48.777 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:48.777 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:48.778 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:08:48.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:08:48.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:48.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:08:49.278 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:50.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:50.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:50.280 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:08:50.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:08:50.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:50.281 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:08:50.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:51.779 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:51.780 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:51.780 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:08:51.780 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:08:51.780 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:51.780 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:08:52.204 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:53.206 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:53.207 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:53.207 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:08:53.207 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:08:53.207 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:53.207 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:08:53.698 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:54.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:54.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:54.701 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:08:54.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:08:54.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:54.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:08:55.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:56.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:56.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:56.307 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:08:56.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:08:56.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:56.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:08:57.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:58.088 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:58.088 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:58.088 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:08:58.089 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:08:58.089 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:58.089 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:08:58.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:08:59.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:08:59.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:08:59.712 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:08:59.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:08:59.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:08:59.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:09:00.688 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:01.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:01.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:01.690 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:09:01.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:09:01.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:01.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:09:02.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:03.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:03.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:03.452 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:09:03.452 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:09:03.452 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:03.452 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:09:04.141 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:05.143 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:05.143 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:05.143 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:09:05.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:09:05.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:05.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:09:05.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:06.831 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:06.831 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:06.831 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:09:06.832 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:09:06.832 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:06.832 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:09:07.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:08.615 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:08.615 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:08.615 INFO:tasks.workunit.client.0.smithi005.stdout:19 2024-05-15T13:09:08.616 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-15T13:09:08.616 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:08.616 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:09:09.054 INFO:tasks.workunit.client.0.smithi005.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3983612568,v1:127.0.0.1:6835/3983612568] [v2:127.0.0.1:6836/3983612568,v1:127.0.0.1:6837/3983612568] exists,up 2760e0d4-4a20-4d2c-86be-e3faa5108f12 2024-05-15T13:09:09.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:09:09.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:09:09.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:09:09.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:09:09.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-05-15T13:09:09.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:09:09.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:09:09.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-05-15T13:09:09.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:09:09.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-05-15T13:09:09.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-15T13:09:09.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:09:09.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:09:09.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:09:09.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-05-15T13:09:09.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-05-15T13:09:09.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:09:09.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:09:09.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:09:09.058 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:09:09.058 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:09:09.058 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:09:09.060 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:09:09.060 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:09:09.060 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:09:09.060 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:09:09.060 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:09:09.060 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:09:09.060 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:09:09.060 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:09:09.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:09:09.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:09:09.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:09:09.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:09:09.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:09:09.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:09:09.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:09:09.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:09:09.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-05-15T13:09:09.061 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:09:09.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c6da9ef7-1016-4005-ac72-2630771407f5 2024-05-15T13:09:09.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 c6da9ef7-1016-4005-ac72-2630771407f5' 2024-05-15T13:09:09.062 INFO:tasks.workunit.client.0.smithi005.stdout:add osd5 c6da9ef7-1016-4005-ac72-2630771407f5 2024-05-15T13:09:09.062 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:09:09.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD1s0RmbH/+BRAAJTuuRO6+Prc6BL1/IGHMtA== 2024-05-15T13:09:09.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD1s0RmbH/+BRAAJTuuRO6+Prc6BL1/IGHMtA=="}' 2024-05-15T13:09:09.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c6da9ef7-1016-4005-ac72-2630771407f5 -i td/test-erasure-eio/5/new.json 2024-05-15T13:09:09.916 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:09:09.935 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-05-15T13:09:09.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD1s0RmbH/+BRAAJTuuRO6+Prc6BL1/IGHMtA== --osd-uuid c6da9ef7-1016-4005-ac72-2630771407f5 2024-05-15T13:09:09.977 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:09.987+0000 7f6dafc8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:09:10.048 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:10.059+0000 7f6dafc8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:09:10.049 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:10.059+0000 7f6dafc8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:09:10.050 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:10.059+0000 7f6dafc8c7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-15T13:09:10.050 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:10.059+0000 7f6dafc8c7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-15T13:09:10.050 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:10.059+0000 7f6dafc8c7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-15T13:09:10.164 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:10.175+0000 7f6dafc8c7c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-05-15T13:09:18.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-05-15T13:09:18.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:09:18.113 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd5 key to auth repository 2024-05-15T13:09:18.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-05-15T13:09:18.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:09:18.685 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.5 2024-05-15T13:09:18.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-05-15T13:09:18.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:09:18.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:09:18.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:09:18.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:09:18.716 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:18.727+0000 7f0c5ab5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:09:18.718 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:18.727+0000 7f0c5ab5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:09:18.719 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:18.731+0000 7f0c5ab5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:09:19.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-05-15T13:09:19.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:09:19.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-05-15T13:09:19.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:09:19.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:09:19.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:19.125 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:09:19.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:09:19.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:19.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:09:19.260 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:19.271+0000 7f0c5ab5b7c0 -1 Falling back to public interface 2024-05-15T13:09:19.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:20.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:20.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:20.537 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:09:20.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:09:20.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:20.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:09:20.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:21.897 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:21.907+0000 7f0c5ab5b7c0 -1 osd.5 0 log_to_monitors true 2024-05-15T13:09:21.977 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:21.977 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:21.977 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:09:21.977 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:09:21.978 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:21.978 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:09:22.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:23.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:23.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:23.729 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:09:23.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:09:23.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:23.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:09:24.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:25.508 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:25.508 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:25.508 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:09:25.509 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:09:25.509 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:25.509 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:09:26.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:27.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:27.087 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:27.088 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:09:27.088 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:09:27.088 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:27.088 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:09:27.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:28.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:28.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:28.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:09:28.615 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:09:28.615 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:28.615 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:09:29.302 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:30.303 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:30.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:30.304 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:09:30.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:09:30.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:30.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:09:30.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:31.932 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:31.932 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:31.932 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:09:31.932 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:09:31.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:31.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:09:32.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:33.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:33.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:33.361 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:09:33.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:09:33.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:33.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:09:33.903 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:34.905 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:34.905 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:34.906 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:09:34.906 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:09:34.906 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:34.906 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:09:35.404 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:36.406 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:36.406 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:36.406 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:09:36.407 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:09:36.407 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:36.407 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:09:37.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:38.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:38.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:38.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:09:38.312 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:09:38.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:38.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:09:38.997 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:39.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:39.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:39.999 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:09:40.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:09:40.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:40.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:09:40.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:41.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:41.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:41.706 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:09:41.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:09:41.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:41.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:09:42.187 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:43.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:43.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:43.189 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:09:43.190 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:09:43.190 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:43.190 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:09:43.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:44.856 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:44.856 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:44.856 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:09:44.857 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:09:44.857 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:44.857 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:09:46.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:47.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:47.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:47.171 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:09:47.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:09:47.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:47.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:09:47.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:48.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:48.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:48.764 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:09:48.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:09:48.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:48.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:09:49.301 INFO:tasks.workunit.client.0.smithi005.stdout:osd.5 up in weight 1 up_from 51 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3635501569,v1:127.0.0.1:6843/3635501569] [v2:127.0.0.1:6844/3635501569,v1:127.0.0.1:6845/3635501569] exists,up c6da9ef7-1016-4005-ac72-2630771407f5 2024-05-15T13:09:49.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:09:49.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:09:49.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:09:49.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:09:49.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-05-15T13:09:49.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:09:49.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:09:49.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-05-15T13:09:49.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:09:49.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-05-15T13:09:49.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-15T13:09:49.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:09:49.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:09:49.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:09:49.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-05-15T13:09:49.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-05-15T13:09:49.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:09:49.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:09:49.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:09:49.306 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:09:49.306 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:09:49.306 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:09:49.307 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:09:49.307 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:09:49.307 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:09:49.307 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:09:49.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:09:49.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:09:49.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:09:49.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:09:49.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:09:49.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:09:49.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:09:49.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:09:49.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:09:49.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:09:49.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:09:49.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:09:49.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-05-15T13:09:49.308 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:09:49.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a6b35c88-e287-43aa-b60e-5eaf8802bcd6 2024-05-15T13:09:49.308 INFO:tasks.workunit.client.0.smithi005.stdout:add osd6 a6b35c88-e287-43aa-b60e-5eaf8802bcd6 2024-05-15T13:09:49.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 a6b35c88-e287-43aa-b60e-5eaf8802bcd6' 2024-05-15T13:09:49.309 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:09:49.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAdtERmasjJFBAA5hcDVMEw77cSAeSvQKm4UA== 2024-05-15T13:09:49.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAdtERmasjJFBAA5hcDVMEw77cSAeSvQKm4UA=="}' 2024-05-15T13:09:49.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a6b35c88-e287-43aa-b60e-5eaf8802bcd6 -i td/test-erasure-eio/6/new.json 2024-05-15T13:09:50.156 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:09:50.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-05-15T13:09:50.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAdtERmasjJFBAA5hcDVMEw77cSAeSvQKm4UA== --osd-uuid a6b35c88-e287-43aa-b60e-5eaf8802bcd6 2024-05-15T13:09:50.212 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:50.223+0000 7f71e1b497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:09:50.256 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:50.267+0000 7f71e1b497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:09:50.258 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:50.267+0000 7f71e1b497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:09:50.258 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:50.267+0000 7f71e1b497c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-15T13:09:50.258 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:50.267+0000 7f71e1b497c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-15T13:09:50.259 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:50.267+0000 7f71e1b497c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-15T13:09:50.400 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:50.411+0000 7f71e1b497c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-05-15T13:09:56.885 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-05-15T13:09:56.885 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:09:56.886 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-05-15T13:09:56.887 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd6 key to auth repository 2024-05-15T13:09:56.887 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:09:57.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-05-15T13:09:57.451 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.6 2024-05-15T13:09:57.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:09:57.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:09:57.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:09:57.452 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:09:57.481 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:57.490+0000 7fe2a9fad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:09:57.507 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:57.514+0000 7fe2a9fad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:09:57.508 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:57.518+0000 7fe2a9fad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:09:57.891 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-05-15T13:09:57.891 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:09:57.891 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-05-15T13:09:57.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:09:57.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:09:57.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:57.892 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:09:57.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:09:57.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:57.893 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:09:58.069 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:09:58.078+0000 7fe2a9fad7c0 -1 Falling back to public interface 2024-05-15T13:09:58.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:09:59.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:09:59.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:09:59.308 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:09:59.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:09:59.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:09:59.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:09:59.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:10:00.508 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:10:00.518+0000 7fe2a9fad7c0 -1 osd.6 0 log_to_monitors true 2024-05-15T13:10:00.741 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:10:00.741 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:10:00.741 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:10:00.741 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:10:00.742 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:10:00.742 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:10:01.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:10:02.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:10:02.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:10:02.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:10:02.319 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:10:02.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:10:02.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:10:03.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:10:04.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:10:04.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:10:04.320 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:10:04.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:10:04.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:10:04.321 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:10:04.868 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:10:05.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:10:05.870 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:10:05.870 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:10:05.870 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:10:05.870 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:10:05.870 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:10:06.442 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:10:07.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:10:07.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:10:07.444 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:10:07.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:10:07.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:10:07.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:10:07.904 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:10:08.906 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:10:08.906 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:10:08.906 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:10:08.907 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:10:08.907 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:10:08.907 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:10:09.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:10:10.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:10:10.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:10:10.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:10:10.489 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:10:10.490 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:10:10.490 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:10:11.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:10:12.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:10:12.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:10:12.018 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:10:12.019 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:10:12.019 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:10:12.019 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:10:12.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:10:13.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:10:13.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:10:13.560 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:10:13.561 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:10:13.561 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:10:13.561 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:10:14.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:10:15.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:10:15.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:10:15.171 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:10:15.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:10:15.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:10:15.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:10:16.324 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:10:17.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:10:17.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:10:17.326 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:10:17.326 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:10:17.326 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:10:17.326 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:10:18.053 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:10:19.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:10:19.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:10:19.055 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:10:19.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:10:19.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:10:19.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:10:19.947 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:10:20.949 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:10:20.949 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:10:20.949 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:10:20.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:10:20.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:10:20.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:10:21.932 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:10:22.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:10:22.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:10:22.934 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:10:22.935 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:10:22.935 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:10:22.935 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:10:24.032 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:10:25.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:10:25.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:10:25.035 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:10:25.035 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:10:25.035 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:10:25.035 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:10:25.652 INFO:tasks.workunit.client.0.smithi005.stdout:osd.6 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3826931486,v1:127.0.0.1:6851/3826931486] [v2:127.0.0.1:6852/3826931486,v1:127.0.0.1:6853/3826931486] exists,up a6b35c88-e287-43aa-b60e-5eaf8802bcd6 2024-05-15T13:10:25.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:10:25.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:10:25.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:10:25.653 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-15T13:10:25.653 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T13:10:25.653 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T13:10:25.654 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:10:25.654 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:10:25.654 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:10:25.654 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.0.asok 2024-05-15T13:10:25.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-15T13:10:25.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-osd.0.asok log flush 2024-05-15T13:10:25.771 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T13:10:25.784 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-15T13:10:25.792 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T13:06:03.755+0000 7f5d319057c0 0 load: jerasure load: lrc 2024-05-15T13:10:25.793 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:486: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS='--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:10:25.793 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:488: TEST_ec_recovery_multiple_objects_eio: local poolname=pool-jerasure 2024-05-15T13:10:25.793 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:489: TEST_ec_recovery_multiple_objects_eio: create_erasure_coded_pool pool-jerasure 3 2 2024-05-15T13:10:25.793 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-15T13:10:25.793 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-15T13:10:25.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-05-15T13:10:25.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-15T13:10:25.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-05-15T13:10:25.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-15T13:10:25.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-05-15T13:10:27.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-15T13:10:27.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-15T13:10:28.590 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-jerasure' already exists 2024-05-15T13:10:28.610 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T13:10:29.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-15T13:10:29.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:10:29.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:10:29.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:10:29.613 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:10:29.614 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:10:29.614 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:10:29.614 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:10:29.614 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:10:29.615 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:10:29.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:10:29.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:10:29.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:10:29.747 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:10:29.747 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:10:29.747 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:10:30.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:10:30.182 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:10:30.182 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:10:30.183 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T13:10:30.183 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T13:10:30.183 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T13:10:30.183 INFO:tasks.workunit.client.0.smithi005.stderr:6' 2024-05-15T13:10:30.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:10:30.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:10:30.183 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:10:30.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803832 2024-05-15T13:10:30.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803832 2024-05-15T13:10:30.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803832' 2024-05-15T13:10:30.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:10:30.382 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:10:30.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574898 2024-05-15T13:10:30.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574898 2024-05-15T13:10:30.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803832 1-55834574898' 2024-05-15T13:10:30.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:10:30.546 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:10:30.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=77309411367 2024-05-15T13:10:30.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 77309411367 2024-05-15T13:10:30.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803832 1-55834574898 2-77309411367' 2024-05-15T13:10:30.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:10:30.703 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:10:30.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084322 2024-05-15T13:10:30.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084322 2024-05-15T13:10:30.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803832 1-55834574898 2-77309411367 3-120259084322' 2024-05-15T13:10:30.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:10:30.864 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T13:10:31.030 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691863 2024-05-15T13:10:31.030 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691863 2024-05-15T13:10:31.030 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803832 1-55834574898 2-77309411367 3-120259084322 4-171798691863' 2024-05-15T13:10:31.030 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:10:31.031 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-15T13:10:31.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=219043332111 2024-05-15T13:10:31.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 219043332111 2024-05-15T13:10:31.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803832 1-55834574898 2-77309411367 3-120259084322 4-171798691863 5-219043332111' 2024-05-15T13:10:31.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:10:31.268 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-15T13:10:31.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005065 2024-05-15T13:10:31.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005065 2024-05-15T13:10:31.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803832 1-55834574898 2-77309411367 3-120259084322 4-171798691863 5-219043332111 6-261993005065' 2024-05-15T13:10:31.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:10:31.434 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803832 2024-05-15T13:10:31.434 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:10:31.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:10:31.436 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803832 2024-05-15T13:10:31.437 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:10:31.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803832 2024-05-15T13:10:31.438 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 25769803832 2024-05-15T13:10:31.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803832' 2024-05-15T13:10:31.439 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:10:31.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803832 -lt 25769803832 2024-05-15T13:10:31.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:10:31.878 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574898 2024-05-15T13:10:31.878 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:10:31.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:10:31.881 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574898 2024-05-15T13:10:31.881 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:10:31.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574898 2024-05-15T13:10:31.883 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 55834574898 2024-05-15T13:10:31.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574898' 2024-05-15T13:10:31.883 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:10:32.302 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574898 -lt 55834574898 2024-05-15T13:10:32.302 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:10:32.303 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-77309411367 2024-05-15T13:10:32.303 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:10:32.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:10:32.305 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-77309411367 2024-05-15T13:10:32.305 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:10:32.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=77309411367 2024-05-15T13:10:32.306 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 77309411367 2024-05-15T13:10:32.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 77309411367' 2024-05-15T13:10:32.307 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:10:32.716 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411367 -lt 77309411367 2024-05-15T13:10:32.716 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:10:32.717 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:10:32.717 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084322 2024-05-15T13:10:32.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:10:32.720 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084322 2024-05-15T13:10:32.720 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:10:32.721 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 120259084322 2024-05-15T13:10:32.721 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084322 2024-05-15T13:10:32.721 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084322' 2024-05-15T13:10:32.721 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:10:33.156 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084323 -lt 120259084322 2024-05-15T13:10:33.156 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:10:33.156 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-171798691863 2024-05-15T13:10:33.157 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:10:33.158 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T13:10:33.159 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-171798691863 2024-05-15T13:10:33.159 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:10:33.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691863 2024-05-15T13:10:33.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 171798691863' 2024-05-15T13:10:33.161 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.4 seq 171798691863 2024-05-15T13:10:33.161 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T13:10:33.596 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691863 -lt 171798691863 2024-05-15T13:10:33.597 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:10:33.597 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-219043332111 2024-05-15T13:10:33.597 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:10:33.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-15T13:10:33.600 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-219043332111 2024-05-15T13:10:33.601 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:10:33.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=219043332111 2024-05-15T13:10:33.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 219043332111' 2024-05-15T13:10:33.602 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.5 seq 219043332111 2024-05-15T13:10:33.602 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-15T13:10:34.022 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 219043332111 -lt 219043332111 2024-05-15T13:10:34.022 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:10:34.022 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-261993005065 2024-05-15T13:10:34.023 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:10:34.024 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-15T13:10:34.025 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:10:34.025 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-261993005065 2024-05-15T13:10:34.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005065 2024-05-15T13:10:34.026 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.6 seq 261993005065 2024-05-15T13:10:34.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 261993005065' 2024-05-15T13:10:34.027 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-15T13:10:34.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005065 -lt 261993005065 2024-05-15T13:10:34.451 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:10:34.452 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:10:34.452 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:10:35.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:10:35.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:10:35.038 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:10:35.038 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:10:35.039 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:10:35.039 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:10:35.039 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:10:35.039 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:10:35.479 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:10:35.480 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:10:35.480 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:10:35.480 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:10:36.033 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:10:36.033 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:10:36.033 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:10:36.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:491: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test1 2024-05-15T13:10:36.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-15T13:10:36.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-15T13:10:36.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2024-05-15T13:10:36.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:10:36.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-15T13:10:36.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:10:36.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-15T13:10:36.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:10:36.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-15T13:10:36.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:10:36.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-15T13:10:36.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2024-05-15T13:10:36.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:492: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test2 2024-05-15T13:10:36.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-15T13:10:36.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-15T13:10:36.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2024-05-15T13:10:36.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:10:36.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-15T13:10:36.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:10:36.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-15T13:10:36.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:10:36.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-15T13:10:36.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:10:36.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-15T13:10:36.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2024-05-15T13:10:36.637 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:493: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test3 2024-05-15T13:10:36.637 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-15T13:10:36.637 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-15T13:10:36.637 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2024-05-15T13:10:36.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:10:36.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-15T13:10:36.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:10:36.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-15T13:10:36.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:10:36.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-15T13:10:36.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:10:36.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-15T13:10:36.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2024-05-15T13:10:36.997 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:496: TEST_ec_recovery_multiple_objects_eio: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-05-15T13:10:36.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-15T13:10:36.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-15T13:10:36.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-15T13:10:36.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-15T13:10:36.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-15T13:10:36.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-15T13:10:36.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-05-15T13:10:36.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-15T13:10:36.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-15T13:10:36.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-15T13:10:36.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-05-15T13:10:36.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-15T13:10:36.999 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-05-15T13:10:36.999 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T13:10:36.999 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-05-15T13:10:36.999 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-05-15T13:10:36.999 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T13:10:37.433 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T13:10:37.433 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T13:10:37.434 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:10:37.434 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T13:10:37.434 INFO:tasks.workunit.client.0.smithi005.stderr:6' 2024-05-15T13:10:37.434 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-05-15T13:10:37.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2024-05-15T13:10:37.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-15T13:10:37.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-05-15T13:10:37.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-15T13:10:37.435 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-05-15T13:10:37.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-15T13:10:37.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-05-15T13:10:37.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-15T13:10:37.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-05-15T13:10:37.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-15T13:10:37.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-15T13:10:37.436 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-15T13:10:37.437 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-05-15T13:10:37.437 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-15T13:10:37.437 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-15T13:10:37.438 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:10:37.438 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:10:37.438 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:10:37.438 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.3.asok 2024-05-15T13:10:37.438 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.80702/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-05-15T13:10:37.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-15T13:10:37.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-15T13:10:37.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-15T13:10:37.570 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-05-15T13:10:37.570 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-15T13:10:37.570 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-15T13:10:37.571 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:10:37.571 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:10:37.571 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:10:37.571 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.3.asok 2024-05-15T13:10:37.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-15T13:10:37.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-05-15T13:10:37.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:497: TEST_ec_recovery_multiple_objects_eio: ceph osd out 0 2024-05-15T13:10:38.244 INFO:tasks.workunit.client.0.smithi005.stderr:osd.0 is already out. 2024-05-15T13:10:38.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:500: TEST_ec_recovery_multiple_objects_eio: wait_for_clean 2024-05-15T13:10:38.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:10:38.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:10:38.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:10:38.264 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:10:38.265 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:10:38.265 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:10:38.265 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:10:38.266 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:10:38.266 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:10:38.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:10:38.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:10:38.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:10:38.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:10:38.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:10:38.382 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:10:39.204 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:10:39.204 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:10:39.204 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:10:39.204 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T13:10:39.205 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T13:10:39.205 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T13:10:39.205 INFO:tasks.workunit.client.0.smithi005.stderr:6' 2024-05-15T13:10:39.205 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:10:39.205 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:10:39.205 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:10:39.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803836 2024-05-15T13:10:39.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803836 2024-05-15T13:10:39.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803836' 2024-05-15T13:10:39.437 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:10:39.437 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:10:39.682 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574902 2024-05-15T13:10:39.683 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574902 2024-05-15T13:10:39.683 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803836 1-55834574902' 2024-05-15T13:10:39.683 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:10:39.683 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:10:39.846 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=77309411371 2024-05-15T13:10:39.846 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 77309411371 2024-05-15T13:10:39.846 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803836 1-55834574902 2-77309411371' 2024-05-15T13:10:39.846 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:10:39.846 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:10:40.080 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084326 2024-05-15T13:10:40.080 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084326 2024-05-15T13:10:40.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803836 1-55834574902 2-77309411371 3-120259084326' 2024-05-15T13:10:40.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:10:40.081 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T13:10:40.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691867 2024-05-15T13:10:40.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691867 2024-05-15T13:10:40.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803836 1-55834574902 2-77309411371 3-120259084326 4-171798691867' 2024-05-15T13:10:40.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:10:40.244 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-15T13:10:40.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=219043332115 2024-05-15T13:10:40.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 219043332115 2024-05-15T13:10:40.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803836 1-55834574902 2-77309411371 3-120259084326 4-171798691867 5-219043332115' 2024-05-15T13:10:40.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:10:40.408 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-15T13:10:40.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005068 2024-05-15T13:10:40.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005068 2024-05-15T13:10:40.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803836 1-55834574902 2-77309411371 3-120259084326 4-171798691867 5-219043332115 6-261993005068' 2024-05-15T13:10:40.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:10:40.565 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803836 2024-05-15T13:10:40.565 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:10:40.567 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:10:40.568 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:10:40.568 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803836 2024-05-15T13:10:40.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803836 2024-05-15T13:10:40.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803836' 2024-05-15T13:10:40.570 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 25769803836 2024-05-15T13:10:40.570 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:10:40.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803836 -lt 25769803836 2024-05-15T13:10:40.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:10:40.982 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574902 2024-05-15T13:10:40.982 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:10:40.984 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:10:40.984 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574902 2024-05-15T13:10:40.985 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:10:40.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574902 2024-05-15T13:10:40.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574902' 2024-05-15T13:10:40.986 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 55834574902 2024-05-15T13:10:40.986 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:10:41.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574902 -lt 55834574902 2024-05-15T13:10:41.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:10:41.430 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-77309411371 2024-05-15T13:10:41.430 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:10:41.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:10:41.433 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-77309411371 2024-05-15T13:10:41.433 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:10:41.434 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=77309411371 2024-05-15T13:10:41.434 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 77309411371 2024-05-15T13:10:41.434 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 77309411371' 2024-05-15T13:10:41.434 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:10:41.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411371 -lt 77309411371 2024-05-15T13:10:41.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:10:41.928 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084326 2024-05-15T13:10:41.928 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:10:41.929 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:10:41.930 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084326 2024-05-15T13:10:41.930 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:10:41.932 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084326 2024-05-15T13:10:41.932 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084326' 2024-05-15T13:10:41.932 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 120259084326 2024-05-15T13:10:41.932 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:10:42.420 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084326 -lt 120259084326 2024-05-15T13:10:42.420 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:10:42.421 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-171798691867 2024-05-15T13:10:42.421 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:10:42.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T13:10:42.424 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:10:42.424 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-171798691867 2024-05-15T13:10:42.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691867 2024-05-15T13:10:42.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 171798691867' 2024-05-15T13:10:42.425 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.4 seq 171798691867 2024-05-15T13:10:42.426 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T13:10:42.842 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691867 -lt 171798691867 2024-05-15T13:10:42.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:10:42.843 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-219043332115 2024-05-15T13:10:42.843 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:10:42.845 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-15T13:10:42.846 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-219043332115 2024-05-15T13:10:42.846 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:10:42.847 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=219043332115 2024-05-15T13:10:42.847 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 219043332115' 2024-05-15T13:10:42.847 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.5 seq 219043332115 2024-05-15T13:10:42.847 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-15T13:10:43.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 219043332115 -lt 219043332115 2024-05-15T13:10:43.296 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:10:43.296 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-261993005068 2024-05-15T13:10:43.296 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:10:43.296 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-15T13:10:43.296 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-261993005068 2024-05-15T13:10:43.296 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:10:43.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005068 2024-05-15T13:10:43.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 261993005068' 2024-05-15T13:10:43.297 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.6 seq 261993005068 2024-05-15T13:10:43.298 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-15T13:10:43.736 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005069 -lt 261993005068 2024-05-15T13:10:43.737 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:10:43.737 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:10:43.737 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:10:44.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:10:44.297 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:10:44.298 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:10:44.298 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:10:44.298 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:10:44.298 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:10:44.299 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:10:44.299 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:10:44.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:10:44.735 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:10:44.735 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:10:44.735 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:10:45.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:10:45.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:10:45.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:10:45.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:502: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test1 2024-05-15T13:10:45.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-15T13:10:45.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-15T13:10:45.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2024-05-15T13:10:45.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-15T13:10:45.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-15T13:10:45.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2024-05-15T13:10:45.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-15T13:10:45.423 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-15T13:10:45.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:503: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test2 2024-05-15T13:10:45.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-15T13:10:45.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-15T13:10:45.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2024-05-15T13:10:45.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-15T13:10:45.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-15T13:10:45.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2024-05-15T13:10:45.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-15T13:10:45.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-15T13:10:45.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:504: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test3 2024-05-15T13:10:45.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-15T13:10:45.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-15T13:10:45.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2024-05-15T13:10:45.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-15T13:10:45.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-15T13:10:45.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2024-05-15T13:10:45.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-15T13:10:45.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-15T13:10:45.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:506: TEST_ec_recovery_multiple_objects_eio: delete_erasure_coded_pool pool-jerasure 2024-05-15T13:10:45.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-15T13:10:45.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-15T13:10:45.989 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-jerasure' does not exist 2024-05-15T13:10:46.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-15T13:10:46.583 INFO:tasks.workunit.client.0.smithi005.stderr:erasure-code-profile myprofile does not exist 2024-05-15T13:10:46.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-15T13:10:46.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-15T13:10:46.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T13:10:46.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-15T13:10:46.602 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:10:46.602 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:10:46.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:10:46.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:10:46.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:10:46.914 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:10:46.915 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T13:10:46.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T13:10:46.917 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T13:10:46.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T13:10:46.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T13:10:46.919 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T13:10:46.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:10:46.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T13:10:46.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T13:10:46.921 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:10:46.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:10:46.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T13:10:46.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-15T13:10:46.966 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T13:10:46.966 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:10:46.966 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:10:46.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80702 2024-05-15T13:10:46.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T13:10:46.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T13:10:46.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-15T13:10:46.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-15T13:10:46.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-15T13:10:46.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-15T13:10:46.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-15T13:10:46.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T13:10:46.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-15T13:10:46.969 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:10:46.969 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:10:46.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:10:46.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:10:46.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:10:46.971 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:10:46.972 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T13:10:46.972 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T13:10:46.973 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T13:10:46.974 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T13:10:46.974 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T13:10:46.975 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T13:10:46.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:10:46.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T13:10:46.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T13:10:46.976 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:10:46.977 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:10:46.978 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T13:10:46.978 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-15T13:10:46.980 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T13:10:46.980 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:10:46.980 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:10:46.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80702 2024-05-15T13:10:46.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T13:10:46.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T13:10:46.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-15T13:10:46.983 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T13:10:46.983 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:10:46.983 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:10:46.983 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.80702 2024-05-15T13:10:46.985 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T13:10:46.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-15T13:10:46.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T13:10:46.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-15T13:10:46.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-15T13:10:46.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-15T13:10:46.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T13:10:46.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T13:10:46.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T13:10:46.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-15T13:10:46.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-15T13:10:47.382 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T13:10:47.382 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:10:47.382 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:10:47.382 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:10:47.383 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:10:47.383 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:10:47.383 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:10:47.383 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-15T13:10:47.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T13:10:47.601 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T13:10:47.601 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T13:10:47.601 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T13:10:47.601 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T13:10:47.602 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T13:10:47.602 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T13:10:47.602 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T13:10:47.602 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T13:10:47.603 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:10:47.603 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:10:47.603 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:10:47.603 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T13:10:47.603 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T13:10:47.603 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.80702/ceph-mon.a.asok config get fsid 2024-05-15T13:10:47.720 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T13:10:47.720 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T13:10:47.720 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T13:10:47.720 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T13:10:47.721 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T13:10:47.721 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T13:10:47.721 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T13:10:47.721 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T13:10:47.722 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:10:47.722 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:10:47.722 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:10:47.722 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T13:10:47.722 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T13:10:47.722 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.80702/ceph-mon.a.asok config get mon_host 2024-05-15T13:10:47.833 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-15T13:10:47.833 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-15T13:10:47.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T13:10:47.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T13:10:47.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T13:10:47.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-15T13:10:47.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T13:10:48.775 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T13:10:48.775 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:10:48.775 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:10:48.775 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:10:48.775 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:10:48.775 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:10:48.775 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:10:48.776 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T13:10:48.777 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T13:10:48.840 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-15T13:10:48.840 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-15T13:10:49.146 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd' already exists 2024-05-15T13:10:49.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T13:10:50.165 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-15T13:10:50.166 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T13:10:50.166 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T13:10:50.167 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:10:50.167 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:10:50.167 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:10:50.167 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T13:10:50.167 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-15T13:10:50.167 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-mon.a.asok log flush 2024-05-15T13:10:50.266 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T13:10:50.278 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-15T13:10:50.281 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T13:10:47.434+0000 7ffa8e07ec80 0 load: jerasure load: lrc 2024-05-15T13:10:50.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_unfound td/test-erasure-eio 2024-05-15T13:10:50.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:609: TEST_ec_recovery_unfound: local dir=td/test-erasure-eio 2024-05-15T13:10:50.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:610: TEST_ec_recovery_unfound: local objname=myobject 2024-05-15T13:10:50.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_recovery_unfound: local lastobj=100 2024-05-15T13:10:50.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_recovery_unfound: local testobj=obj75 2024-05-15T13:10:50.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:615: TEST_ec_recovery_unfound: ORIG_ARGS='--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:10:50.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:616: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-15T13:10:50.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-15T13:10:50.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_recovery_unfound: setup_osds 5 2024-05-15T13:10:50.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2024-05-15T13:10:50.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-15T13:10:50.283 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2024-05-15T13:10:50.284 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2024-05-15T13:10:50.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:10:50.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-15T13:10:50.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:10:50.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:10:50.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T13:10:50.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:10:50.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-15T13:10:50.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-15T13:10:50.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:10:50.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:10:50.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:10:50.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-15T13:10:50.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-15T13:10:50.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:10:50.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:10:50.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:10:50.290 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:10:50.290 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:10:50.290 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:10:50.290 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:10:50.290 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:10:50.290 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:10:50.290 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:10:50.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:10:50.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:10:50.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:10:50.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:10:50.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:10:50.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:10:50.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:10:50.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:10:50.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:10:50.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:10:50.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:10:50.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:10:50.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-15T13:10:50.291 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:10:50.291 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 df6b3732-6eb2-49ae-8b8d-9df736582097 2024-05-15T13:10:50.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=df6b3732-6eb2-49ae-8b8d-9df736582097 2024-05-15T13:10:50.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 df6b3732-6eb2-49ae-8b8d-9df736582097' 2024-05-15T13:10:50.292 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:10:50.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBatERmv6GiExAA4gXifUemCYNVxW7BP0eYUA== 2024-05-15T13:10:50.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBatERmv6GiExAA4gXifUemCYNVxW7BP0eYUA=="}' 2024-05-15T13:10:50.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new df6b3732-6eb2-49ae-8b8d-9df736582097 -i td/test-erasure-eio/0/new.json 2024-05-15T13:10:50.639 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:10:50.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-15T13:10:50.656 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBatERmv6GiExAA4gXifUemCYNVxW7BP0eYUA== --osd-uuid df6b3732-6eb2-49ae-8b8d-9df736582097 2024-05-15T13:10:50.695 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:10:50.701+0000 7f3821e8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:10:50.716 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:10:50.725+0000 7f3821e8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:10:50.718 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:10:50.725+0000 7f3821e8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:10:50.718 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:10:50.725+0000 7f3821e8f7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T13:10:50.718 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:10:50.725+0000 7f3821e8f7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T13:10:50.718 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:10:50.725+0000 7f3821e8f7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T13:10:50.756 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:10:50.761+0000 7f3821e8f7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-15T13:10:57.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-15T13:10:57.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:10:57.395 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T13:10:57.395 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T13:10:57.395 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:10:57.984 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T13:10:57.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T13:10:57.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:10:57.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:10:57.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:10:57.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:10:58.027 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:10:58.033+0000 7f69161037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:10:58.056 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:10:58.061+0000 7f69161037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:10:58.058 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:10:58.065+0000 7f69161037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:10:58.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T13:10:58.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:10:58.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T13:10:58.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:10:58.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:10:58.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:10:58.445 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:10:58.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:10:58.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:10:58.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:10:58.642 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:10:58.649+0000 7f69161037c0 -1 Falling back to public interface 2024-05-15T13:10:58.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:10:59.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:10:59.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:10:59.874 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:10:59.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:10:59.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:10:59.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:11:00.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:01.293 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:11:01.301+0000 7f69161037c0 -1 osd.0 0 log_to_monitors true 2024-05-15T13:11:01.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:01.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:01.325 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:11:01.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:11:01.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:01.326 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:11:02.041 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:03.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:03.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:03.043 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:11:03.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:11:03.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:03.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:11:03.493 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:04.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:04.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:04.496 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:11:04.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:11:04.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:04.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:11:04.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:05.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:05.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:05.939 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:11:05.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:11:05.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:05.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:11:06.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:07.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:07.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:07.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:11:07.516 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:11:07.516 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:07.516 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:11:07.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:08.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:08.969 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:11:08.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:08.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:11:08.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:08.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:11:09.466 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:10.468 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:10.468 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:10.469 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:11:10.469 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:11:10.469 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:10.469 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:11:10.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:11.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:11.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:11.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:11:11.940 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:11:11.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:11.941 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:11:12.493 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:13.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:13.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:13.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:11:13.495 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:11:13.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:13.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:11:14.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:15.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:15.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:15.100 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:11:15.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:11:15.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:15.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:11:15.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:16.952 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:16.952 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:16.952 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:11:16.953 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:11:16.953 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:16.953 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:11:17.842 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:18.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:18.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:18.844 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:11:18.845 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:11:18.845 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:18.845 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:11:19.752 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:20.753 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:20.754 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:11:20.754 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:20.754 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:11:20.754 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:20.754 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:11:21.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:22.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:22.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:22.633 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:11:22.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:11:22.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:22.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:11:23.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:24.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:24.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:24.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:11:24.731 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:11:24.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:24.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:11:25.597 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:26.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:26.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:26.599 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:11:26.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:11:26.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:26.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:11:27.099 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3806705582,v1:127.0.0.1:6803/3806705582] [v2:127.0.0.1:6804/3806705582,v1:127.0.0.1:6805/3806705582] exists,up df6b3732-6eb2-49ae-8b8d-9df736582097 2024-05-15T13:11:27.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:11:27.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:11:27.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:11:27.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:11:27.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-15T13:11:27.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:11:27.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:11:27.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-15T13:11:27.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:11:27.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-15T13:11:27.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-15T13:11:27.103 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:11:27.103 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:11:27.103 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:11:27.103 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-15T13:11:27.103 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-15T13:11:27.103 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:11:27.103 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:11:27.103 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:11:27.103 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:11:27.103 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:11:27.103 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:11:27.103 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:11:27.103 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:11:27.103 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:11:27.103 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:11:27.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:11:27.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:11:27.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:11:27.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:11:27.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:11:27.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:11:27.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:11:27.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:11:27.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:11:27.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:11:27.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:11:27.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:11:27.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-15T13:11:27.105 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:11:27.105 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5802f4e2-5b15-4a8e-95e5-6bc863d6cf6e 2024-05-15T13:11:27.105 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5802f4e2-5b15-4a8e-95e5-6bc863d6cf6e' 2024-05-15T13:11:27.105 INFO:tasks.workunit.client.0.smithi005.stdout:add osd1 5802f4e2-5b15-4a8e-95e5-6bc863d6cf6e 2024-05-15T13:11:27.105 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:11:27.131 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB/tERm+HRdCBAAv5GfoFTSCojcb6Yg+j7J9A== 2024-05-15T13:11:27.131 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB/tERm+HRdCBAAv5GfoFTSCojcb6Yg+j7J9A=="}' 2024-05-15T13:11:27.131 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5802f4e2-5b15-4a8e-95e5-6bc863d6cf6e -i td/test-erasure-eio/1/new.json 2024-05-15T13:11:27.934 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:11:27.953 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-15T13:11:27.955 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB/tERm+HRdCBAAv5GfoFTSCojcb6Yg+j7J9A== --osd-uuid 5802f4e2-5b15-4a8e-95e5-6bc863d6cf6e 2024-05-15T13:11:27.999 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:11:28.005+0000 7f18ae0d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:11:28.002 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:11:28.009+0000 7f18ae0d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:11:28.004 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:11:28.009+0000 7f18ae0d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:11:28.004 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:11:28.009+0000 7f18ae0d37c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T13:11:28.004 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:11:28.009+0000 7f18ae0d37c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T13:11:28.004 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:11:28.009+0000 7f18ae0d37c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T13:11:28.100 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:11:28.105+0000 7f18ae0d37c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-15T13:11:34.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-15T13:11:34.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:11:34.660 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd1 key to auth repository 2024-05-15T13:11:34.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-15T13:11:34.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:11:35.241 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.1 2024-05-15T13:11:35.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-15T13:11:35.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:11:35.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:11:35.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:11:35.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:11:35.283 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:11:35.289+0000 7f90441637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:11:35.322 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:11:35.329+0000 7f90441637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:11:35.324 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:11:35.329+0000 7f90441637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:11:35.693 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-15T13:11:35.693 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:11:35.693 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:11:35.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-15T13:11:35.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:11:35.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:11:35.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:35.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:11:35.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:11:35.694 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:35.911 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:11:35.917+0000 7f90441637c0 -1 Falling back to public interface 2024-05-15T13:11:36.136 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:37.137 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:37.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:37.138 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:11:37.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:11:37.139 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:37.139 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:11:37.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:38.573 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:38.573 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:38.573 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:11:38.574 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:11:38.574 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:38.574 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:11:38.602 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:11:38.609+0000 7f90441637c0 -1 osd.1 0 log_to_monitors true 2024-05-15T13:11:39.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:40.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:40.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:40.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:11:40.185 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:11:40.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:40.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:11:40.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:41.759 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:41.759 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:41.759 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:11:41.759 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:11:41.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:41.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:11:42.248 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:43.249 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:43.249 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:43.250 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:11:43.250 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:11:43.250 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:43.250 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:11:43.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:44.777 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:44.777 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:44.777 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:11:44.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:11:44.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:44.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:11:45.231 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:46.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:46.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:46.234 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:11:46.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:11:46.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:46.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:11:46.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:47.759 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:47.759 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:47.760 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:11:47.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:11:47.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:47.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:11:48.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:49.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:49.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:49.311 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:11:49.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:11:49.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:49.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:11:49.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:50.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:50.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:50.926 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:11:50.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:11:50.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:50.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:11:51.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:52.551 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:52.551 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:52.551 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:11:52.552 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:11:52.552 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:52.552 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:11:53.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:54.092 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:54.093 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:54.093 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:11:54.093 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:11:54.094 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:54.094 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:11:55.457 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:56.459 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:56.459 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:56.459 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:11:56.460 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:11:56.460 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:56.460 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:11:57.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:11:58.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:11:58.339 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:11:58.340 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:11:58.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:11:58.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:11:58.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:11:59.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:00.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:00.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:00.272 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:12:00.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:12:00.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:00.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:12:00.866 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:01.867 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:01.868 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:01.868 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:12:01.868 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:12:01.868 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:01.868 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:12:02.698 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:03.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:03.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:03.701 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:12:03.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:12:03.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:03.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:12:04.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:05.301 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:05.301 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:05.301 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:12:05.302 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:12:05.302 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:05.302 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:12:05.745 INFO:tasks.workunit.client.0.smithi005.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3812018924,v1:127.0.0.1:6811/3812018924] [v2:127.0.0.1:6812/3812018924,v1:127.0.0.1:6813/3812018924] exists,up 5802f4e2-5b15-4a8e-95e5-6bc863d6cf6e 2024-05-15T13:12:05.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:12:05.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:12:05.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:12:05.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:12:05.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-15T13:12:05.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:12:05.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:12:05.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-15T13:12:05.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:12:05.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-15T13:12:05.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-15T13:12:05.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:12:05.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:12:05.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:12:05.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-15T13:12:05.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-15T13:12:05.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:12:05.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:12:05.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:12:05.749 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:12:05.749 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:12:05.749 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:12:05.749 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:12:05.749 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:12:05.750 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:12:05.750 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:12:05.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:12:05.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:12:05.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:12:05.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:12:05.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:12:05.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:12:05.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:12:05.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:12:05.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:12:05.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:12:05.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:12:05.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:12:05.751 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-15T13:12:05.751 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:12:05.751 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c32adb32-5c52-4405-97ca-aa0f50e78237 2024-05-15T13:12:05.752 INFO:tasks.workunit.client.0.smithi005.stdout:add osd2 c32adb32-5c52-4405-97ca-aa0f50e78237 2024-05-15T13:12:05.752 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c32adb32-5c52-4405-97ca-aa0f50e78237' 2024-05-15T13:12:05.752 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:12:05.774 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCltERm8MCvLhAAnYHLR37Pvbz/SdInXYLyCg== 2024-05-15T13:12:05.774 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCltERm8MCvLhAAnYHLR37Pvbz/SdInXYLyCg=="}' 2024-05-15T13:12:05.774 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c32adb32-5c52-4405-97ca-aa0f50e78237 -i td/test-erasure-eio/2/new.json 2024-05-15T13:12:06.362 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:12:06.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-15T13:12:06.379 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCltERm8MCvLhAAnYHLR37Pvbz/SdInXYLyCg== --osd-uuid c32adb32-5c52-4405-97ca-aa0f50e78237 2024-05-15T13:12:06.424 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:06.428+0000 7f87d57b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:12:06.460 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:06.464+0000 7f87d57b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:12:06.462 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:06.468+0000 7f87d57b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:12:06.462 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:06.468+0000 7f87d57b37c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T13:12:06.462 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:06.468+0000 7f87d57b37c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T13:12:06.463 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:06.468+0000 7f87d57b37c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T13:12:06.499 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:06.504+0000 7f87d57b37c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-15T13:12:13.434 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-15T13:12:13.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:12:13.436 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd2 key to auth repository 2024-05-15T13:12:13.437 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-15T13:12:13.437 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:12:13.998 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.2 2024-05-15T13:12:14.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-15T13:12:14.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:12:14.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:12:14.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:12:14.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:12:14.034 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:14.040+0000 7f464dfc37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:12:14.066 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:14.072+0000 7f464dfc37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:12:14.067 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:14.072+0000 7f464dfc37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:12:14.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-15T13:12:14.487 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:12:14.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-15T13:12:14.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:12:14.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:12:14.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:14.488 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:12:14.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:12:14.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:14.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:12:14.639 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:14.644+0000 7f464dfc37c0 -1 Falling back to public interface 2024-05-15T13:12:14.954 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:15.955 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:15.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:15.956 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:12:15.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:12:15.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:15.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:12:16.395 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:17.208 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:17.212+0000 7f464dfc37c0 -1 osd.2 0 log_to_monitors true 2024-05-15T13:12:17.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:17.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:17.397 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:12:17.398 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:12:17.398 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:17.398 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:12:18.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:19.017 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:19.017 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:19.017 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:12:19.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:12:19.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:19.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:12:19.482 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:20.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:20.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:20.484 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:12:20.484 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:12:20.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:20.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:12:20.932 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:21.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:21.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:21.934 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:12:21.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:12:21.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:21.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:12:22.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:23.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:23.392 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:23.392 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:12:23.392 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:12:23.392 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:23.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:12:23.914 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:24.915 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:24.915 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:24.916 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:12:24.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:12:24.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:24.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:12:25.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:26.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:26.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:26.472 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:12:26.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:12:26.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:26.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:12:26.915 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:27.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:27.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:27.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:12:27.917 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:12:27.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:27.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:12:28.524 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:29.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:29.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:29.527 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:12:29.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:12:29.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:29.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:12:30.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:31.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:31.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:31.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:12:31.185 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:12:31.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:31.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:12:31.642 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:32.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:32.644 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:32.644 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:12:32.644 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:12:32.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:32.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:12:33.284 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:34.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:34.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:34.285 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:12:34.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:12:34.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:34.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:12:35.385 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:36.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:36.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:36.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:12:36.388 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:12:36.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:36.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:12:37.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:38.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:38.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:38.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:12:38.267 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:12:38.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:38.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:12:39.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:40.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:40.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:40.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:12:40.272 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:12:40.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:40.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:12:41.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:42.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:42.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:42.311 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:12:42.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:12:42.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:42.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:12:42.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:43.935 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:43.936 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:43.936 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:12:43.936 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:12:43.936 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:43.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:12:44.443 INFO:tasks.workunit.client.0.smithi005.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/275005916,v1:127.0.0.1:6819/275005916] [v2:127.0.0.1:6820/275005916,v1:127.0.0.1:6821/275005916] exists,up c32adb32-5c52-4405-97ca-aa0f50e78237 2024-05-15T13:12:44.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:12:44.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:12:44.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:12:44.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:12:44.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-15T13:12:44.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:12:44.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:12:44.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-15T13:12:44.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:12:44.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-15T13:12:44.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-15T13:12:44.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:12:44.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:12:44.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:12:44.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-15T13:12:44.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-15T13:12:44.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:12:44.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:12:44.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:12:44.449 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:12:44.449 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:12:44.449 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:12:44.449 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:12:44.449 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:12:44.449 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:12:44.450 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:12:44.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:12:44.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:12:44.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:12:44.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:12:44.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:12:44.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:12:44.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:12:44.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:12:44.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:12:44.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:12:44.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:12:44.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:12:44.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-15T13:12:44.451 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:12:44.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=65effdf5-fd4b-411c-97c5-3dace783c16a 2024-05-15T13:12:44.452 INFO:tasks.workunit.client.0.smithi005.stdout:add osd3 65effdf5-fd4b-411c-97c5-3dace783c16a 2024-05-15T13:12:44.452 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 65effdf5-fd4b-411c-97c5-3dace783c16a' 2024-05-15T13:12:44.452 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:12:44.476 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDMtERmC7nVHBAAXD6XA9c3tB/jjRkRjiObuA== 2024-05-15T13:12:44.476 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDMtERmC7nVHBAAXD6XA9c3tB/jjRkRjiObuA=="}' 2024-05-15T13:12:44.476 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 65effdf5-fd4b-411c-97c5-3dace783c16a -i td/test-erasure-eio/3/new.json 2024-05-15T13:12:45.145 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:12:45.161 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-15T13:12:45.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDMtERmC7nVHBAAXD6XA9c3tB/jjRkRjiObuA== --osd-uuid 65effdf5-fd4b-411c-97c5-3dace783c16a 2024-05-15T13:12:45.208 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:45.215+0000 7fbbdb9227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:12:45.210 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:45.215+0000 7fbbdb9227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:12:45.211 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:45.219+0000 7fbbdb9227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:12:45.212 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:45.219+0000 7fbbdb9227c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T13:12:45.212 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:45.219+0000 7fbbdb9227c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T13:12:45.212 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:45.219+0000 7fbbdb9227c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T13:12:45.395 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:45.403+0000 7fbbdb9227c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-15T13:12:52.024 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-15T13:12:52.024 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:12:52.026 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd3 key to auth repository 2024-05-15T13:12:52.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-15T13:12:52.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:12:52.600 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.3 2024-05-15T13:12:52.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-15T13:12:52.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:12:52.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:12:52.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:12:52.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:12:52.643 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:52.651+0000 7fa2c52cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:12:52.670 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:52.675+0000 7fa2c52cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:12:52.670 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:52.679+0000 7fa2c52cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:12:53.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-15T13:12:53.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:12:53.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-15T13:12:53.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:12:53.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:12:53.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:53.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:12:53.052 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:12:53.053 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:53.053 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:12:53.224 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:53.231+0000 7fa2c52cc7c0 -1 Falling back to public interface 2024-05-15T13:12:53.517 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:54.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:54.519 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:54.520 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:12:54.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:12:54.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:54.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:12:54.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:55.968 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:55.968 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:55.968 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:12:55.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:12:55.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:55.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:12:56.041 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:12:56.047+0000 7fa2c52cc7c0 -1 osd.3 0 log_to_monitors true 2024-05-15T13:12:56.497 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:57.499 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:57.499 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:57.499 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:12:57.500 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:12:57.500 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:57.500 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:12:58.359 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:12:59.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:12:59.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:12:59.361 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:12:59.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:12:59.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:12:59.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:12:59.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:00.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:00.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:00.854 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:13:00.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:13:00.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:00.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:13:01.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:02.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:02.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:02.327 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:13:02.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:13:02.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:02.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:13:02.835 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:03.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:03.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:03.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:13:03.837 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:13:03.838 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:03.838 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:13:04.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:05.384 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:05.384 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:05.384 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:13:05.385 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:13:05.385 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:05.385 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:13:05.825 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:06.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:06.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:06.827 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:13:06.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:13:06.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:06.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:13:07.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:08.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:08.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:08.360 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:13:08.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:13:08.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:08.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:13:08.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:09.807 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:09.807 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:09.808 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:13:09.808 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:13:09.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:09.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:13:10.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:11.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:11.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:11.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:13:11.331 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:13:11.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:11.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:13:11.777 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:12.779 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:12.779 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:12.779 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:13:12.780 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:13:12.780 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:12.780 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:13:14.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:15.111 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:15.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:15.112 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:13:15.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:13:15.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:15.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:13:15.977 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:16.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:16.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:16.979 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:13:16.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:13:16.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:16.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:13:17.810 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:18.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:18.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:18.812 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:13:18.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:13:18.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:18.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:13:19.647 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:20.648 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:20.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:20.649 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:13:20.650 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:13:20.650 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:20.650 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:13:21.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:22.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:22.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:22.362 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:13:22.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:13:22.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:22.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:13:23.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:24.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:24.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:24.258 INFO:tasks.workunit.client.0.smithi005.stdout:19 2024-05-15T13:13:24.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-15T13:13:24.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:24.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:13:24.917 INFO:tasks.workunit.client.0.smithi005.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/849711278,v1:127.0.0.1:6827/849711278] [v2:127.0.0.1:6828/849711278,v1:127.0.0.1:6829/849711278] exists,up 65effdf5-fd4b-411c-97c5-3dace783c16a 2024-05-15T13:13:24.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:13:24.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:13:24.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:13:24.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:13:24.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-05-15T13:13:24.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:13:24.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:13:24.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-05-15T13:13:24.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:13:24.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-05-15T13:13:24.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-15T13:13:24.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:13:24.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:13:24.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:13:24.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-05-15T13:13:24.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-05-15T13:13:24.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:13:24.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:13:24.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:13:24.922 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:13:24.922 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:13:24.923 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:13:24.923 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:13:24.923 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:13:24.923 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:13:24.923 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:13:24.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:13:24.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:13:24.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:13:24.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:13:24.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:13:24.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:13:24.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:13:24.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:13:24.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:13:24.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:13:24.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:13:24.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:13:24.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-05-15T13:13:24.924 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:13:24.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2bb1d6b1-42c9-4228-9b3a-2ec8a5576e2c 2024-05-15T13:13:24.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 2bb1d6b1-42c9-4228-9b3a-2ec8a5576e2c' 2024-05-15T13:13:24.925 INFO:tasks.workunit.client.0.smithi005.stdout:add osd4 2bb1d6b1-42c9-4228-9b3a-2ec8a5576e2c 2024-05-15T13:13:24.925 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:13:24.947 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0tERmV8XyOBAAFqW/1zeTyXK5Hxjb+dw+PQ== 2024-05-15T13:13:24.948 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0tERmV8XyOBAAFqW/1zeTyXK5Hxjb+dw+PQ=="}' 2024-05-15T13:13:24.948 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2bb1d6b1-42c9-4228-9b3a-2ec8a5576e2c -i td/test-erasure-eio/4/new.json 2024-05-15T13:13:25.654 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:13:25.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-05-15T13:13:25.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD0tERmV8XyOBAAFqW/1zeTyXK5Hxjb+dw+PQ== --osd-uuid 2bb1d6b1-42c9-4228-9b3a-2ec8a5576e2c 2024-05-15T13:13:25.710 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:13:25.714+0000 7eff6d44d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:13:25.933 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:13:25.938+0000 7eff6d44d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:13:25.934 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:13:25.938+0000 7eff6d44d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:13:25.935 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:13:25.938+0000 7eff6d44d7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T13:13:25.935 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:13:25.942+0000 7eff6d44d7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T13:13:25.935 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:13:25.942+0000 7eff6d44d7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T13:13:26.034 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:13:26.038+0000 7eff6d44d7c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-05-15T13:13:33.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-05-15T13:13:33.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:13:33.064 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-05-15T13:13:33.065 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd4 key to auth repository 2024-05-15T13:13:33.065 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:13:33.645 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.4 2024-05-15T13:13:33.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-05-15T13:13:33.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:13:33.646 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:13:33.646 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:13:33.646 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:13:33.687 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:13:33.690+0000 7fca9d0717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:13:33.715 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:13:33.718+0000 7fca9d0717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:13:33.716 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:13:33.722+0000 7fca9d0717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:13:34.088 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-05-15T13:13:34.089 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:13:34.089 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-05-15T13:13:34.089 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:13:34.089 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:13:34.089 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:13:34.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:34.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:13:34.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:34.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:13:34.277 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:13:34.282+0000 7fca9d0717c0 -1 Falling back to public interface 2024-05-15T13:13:34.530 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:35.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:35.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:35.533 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:13:35.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:13:35.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:35.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:13:35.988 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:36.981 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:13:36.986+0000 7fca9d0717c0 -1 osd.4 0 log_to_monitors true 2024-05-15T13:13:36.990 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:36.990 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:36.990 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:13:36.990 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:13:36.990 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:36.991 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:13:37.978 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:38.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:38.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:38.980 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:13:38.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:13:38.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:38.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:13:39.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:40.683 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:40.683 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:40.683 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:13:40.683 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:13:40.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:40.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:13:41.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:42.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:42.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:42.271 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:13:42.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:13:42.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:42.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:13:42.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:43.751 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:43.751 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:43.751 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:13:43.752 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:13:43.752 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:43.752 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:13:44.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:45.455 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:45.456 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:45.456 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:13:45.456 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:13:45.456 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:45.457 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:13:46.118 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:47.120 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:47.120 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:47.121 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:13:47.121 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:13:47.121 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:47.122 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:13:47.577 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:48.578 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:48.578 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:48.578 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:13:48.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:13:48.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:48.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:13:49.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:50.154 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:50.154 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:50.154 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:13:50.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:13:50.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:50.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:13:50.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:51.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:51.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:51.769 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:13:51.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:13:51.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:51.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:13:52.384 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:53.386 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:53.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:53.387 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:13:53.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:13:53.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:53.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:13:53.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:54.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:54.891 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:54.891 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:13:54.891 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:13:54.891 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:54.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:13:55.896 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:56.898 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:56.898 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:56.899 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:13:56.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:13:56.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:56.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:13:57.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:13:58.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:13:58.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:13:58.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:13:58.861 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:13:58.861 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:13:58.861 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:13:59.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:14:00.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:14:00.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:14:00.586 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:14:00.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:14:00.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:14:00.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:14:01.374 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:14:02.376 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:14:02.376 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:14:02.377 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:14:02.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:14:02.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:14:02.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:14:03.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:14:04.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:14:04.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:14:04.008 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:14:04.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:14:04.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:14:04.009 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:14:04.512 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:14:05.514 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:14:05.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:14:05.515 INFO:tasks.workunit.client.0.smithi005.stdout:19 2024-05-15T13:14:05.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-15T13:14:05.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:14:05.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:14:06.193 INFO:tasks.workunit.client.0.smithi005.stdout:osd.4 up in weight 1 up_from 37 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2219516041,v1:127.0.0.1:6835/2219516041] [v2:127.0.0.1:6836/2219516041,v1:127.0.0.1:6837/2219516041] exists,up 2bb1d6b1-42c9-4228-9b3a-2ec8a5576e2c 2024-05-15T13:14:06.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:14:06.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:14:06.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:14:06.194 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-15T13:14:06.194 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T13:14:06.194 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T13:14:06.194 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:14:06.194 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:14:06.195 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:14:06.195 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.0.asok 2024-05-15T13:14:06.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-15T13:14:06.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-osd.0.asok log flush 2024-05-15T13:14:06.315 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T13:14:06.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-15T13:14:06.335 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T13:10:58.653+0000 7f69161037c0 0 load: jerasure load: lrc 2024-05-15T13:14:06.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: CEPH_ARGS='--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:14:06.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:621: TEST_ec_recovery_unfound: local poolname=pool-jerasure 2024-05-15T13:14:06.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:622: TEST_ec_recovery_unfound: create_erasure_coded_pool pool-jerasure 3 2 2024-05-15T13:14:06.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-15T13:14:06.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-15T13:14:06.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-05-15T13:14:06.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-15T13:14:06.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-05-15T13:14:06.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-15T13:14:06.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-05-15T13:14:07.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-15T13:14:07.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-15T13:14:08.550 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-jerasure' already exists 2024-05-15T13:14:08.572 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T13:14:09.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-15T13:14:09.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:14:09.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:14:09.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:14:09.576 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:14:09.577 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:14:09.577 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:14:09.577 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:14:09.577 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:14:09.577 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:14:09.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:14:09.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:14:09.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:14:09.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:14:09.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:14:09.711 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:14:10.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:14:10.171 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:14:10.171 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:14:10.171 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T13:14:10.171 INFO:tasks.workunit.client.0.smithi005.stderr:4' 2024-05-15T13:14:10.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:14:10.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:14:10.172 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:14:10.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803816 2024-05-15T13:14:10.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803816 2024-05-15T13:14:10.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803816' 2024-05-15T13:14:10.472 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:14:10.472 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:14:10.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574880 2024-05-15T13:14:10.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574880 2024-05-15T13:14:10.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803816 1-55834574880' 2024-05-15T13:14:10.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:14:10.635 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:14:10.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378645 2024-05-15T13:14:10.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378645 2024-05-15T13:14:10.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803816 1-55834574880 2-81604378645' 2024-05-15T13:14:10.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:14:10.838 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:14:10.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117006 2024-05-15T13:14:10.993 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117006 2024-05-15T13:14:10.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803816 1-55834574880 2-81604378645 3-115964117006' 2024-05-15T13:14:10.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:14:10.994 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T13:14:11.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789965 2024-05-15T13:14:11.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789965 2024-05-15T13:14:11.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803816 1-55834574880 2-81604378645 3-115964117006 4-158913789965' 2024-05-15T13:14:11.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:14:11.149 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803816 2024-05-15T13:14:11.150 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:14:11.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:14:11.151 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803816 2024-05-15T13:14:11.151 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:14:11.152 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803816 2024-05-15T13:14:11.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803816' 2024-05-15T13:14:11.153 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 25769803816 2024-05-15T13:14:11.153 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:14:11.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803814 -lt 25769803816 2024-05-15T13:14:11.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-15T13:14:12.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-15T13:14:12.570 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:14:12.987 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803816 -lt 25769803816 2024-05-15T13:14:12.987 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:14:12.987 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574880 2024-05-15T13:14:12.988 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:14:12.989 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:14:12.990 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574880 2024-05-15T13:14:12.990 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:14:12.991 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574880 2024-05-15T13:14:12.991 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 55834574880 2024-05-15T13:14:12.992 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574880' 2024-05-15T13:14:12.992 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:14:13.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574880 -lt 55834574880 2024-05-15T13:14:13.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:14:13.423 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:14:13.423 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378645 2024-05-15T13:14:13.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:14:13.425 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378645 2024-05-15T13:14:13.425 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:14:13.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378645 2024-05-15T13:14:13.427 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 81604378645 2024-05-15T13:14:13.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378645' 2024-05-15T13:14:13.428 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:14:13.868 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378645 -lt 81604378645 2024-05-15T13:14:13.868 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:14:13.868 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117006 2024-05-15T13:14:13.869 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:14:13.870 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:14:13.871 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117006 2024-05-15T13:14:13.872 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:14:13.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117006 2024-05-15T13:14:13.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117006' 2024-05-15T13:14:13.873 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 115964117006 2024-05-15T13:14:13.873 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:14:14.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117006 -lt 115964117006 2024-05-15T13:14:14.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:14:14.288 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-158913789965 2024-05-15T13:14:14.288 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:14:14.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T13:14:14.290 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-158913789965 2024-05-15T13:14:14.290 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:14:14.291 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.4 seq 158913789965 2024-05-15T13:14:14.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789965 2024-05-15T13:14:14.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 158913789965' 2024-05-15T13:14:14.292 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T13:14:14.779 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789965 -lt 158913789965 2024-05-15T13:14:14.780 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:14:14.780 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:14:14.780 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:14:15.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:14:15.324 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:14:15.324 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:14:15.325 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:14:15.325 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:14:15.325 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:14:15.325 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:14:15.325 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:14:15.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:14:15.761 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:14:15.761 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:14:15.761 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:14:16.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:14:16.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:14:16.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:14:16.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:624: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-05-15T13:14:16.720 INFO:tasks.workunit.client.0.smithi005.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-05-15T13:14:16.720 INFO:tasks.workunit.client.0.smithi005.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T13:14:10.114170+0000 0'0 45:15 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-05-15T13:14:08.249128+0000 0'0 2024-05-15T13:14:08.249128+0000 0 0 periodic scrub scheduled @ 2024-05-16T18:30:38.016564+0000 0 0 2024-05-15T13:14:16.720 INFO:tasks.workunit.client.0.smithi005.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T13:14:07.449836+0000 0'0 45:21 [4,1,2] 4 [4,1,2] 4 0'0 2024-05-15T13:10:49.046418+0000 0'0 2024-05-15T13:10:49.046418+0000 0 0 periodic scrub scheduled @ 2024-05-16T18:04:12.590896+0000 0 0 2024-05-15T13:14:16.720 INFO:tasks.workunit.client.0.smithi005.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T13:14:07.340473+0000 0'0 44:102 [0,4,1] 0 [0,4,1] 0 0'0 2024-05-15T13:10:49.046418+0000 0'0 2024-05-15T13:10:49.046418+0000 0 0 periodic scrub scheduled @ 2024-05-16T23:50:27.372650+0000 0 0 2024-05-15T13:14:16.720 INFO:tasks.workunit.client.0.smithi005.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T13:14:07.254980+0000 0'0 45:21 [4,3,0] 4 [4,3,0] 4 0'0 2024-05-15T13:10:49.046418+0000 0'0 2024-05-15T13:10:49.046418+0000 0 0 periodic scrub scheduled @ 2024-05-16T23:45:47.610880+0000 0 0 2024-05-15T13:14:16.720 INFO:tasks.workunit.client.0.smithi005.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T13:12:44.701202+0000 0'0 45:73 [1,0,2] 1 [1,0,2] 1 0'0 2024-05-15T13:10:49.046418+0000 0'0 2024-05-15T13:10:49.046418+0000 0 0 periodic scrub scheduled @ 2024-05-16T19:22:33.701207+0000 0 0 2024-05-15T13:14:16.720 INFO:tasks.workunit.client.0.smithi005.stdout: 2024-05-15T13:14:16.721 INFO:tasks.workunit.client.0.smithi005.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-05-15T13:14:16.721 INFO:tasks.workunit.client.0.smithi005.stderr:dumped pgs 2024-05-15T13:14:16.737 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:626: TEST_ec_recovery_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2024-05-15T13:14:16.737 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-15T13:14:16.737 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-15T13:14:16.737 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-05-15T13:14:16.737 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:14:16.737 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-15T13:14:16.737 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:14:16.737 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-15T13:14:16.737 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:14:16.737 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-15T13:14:16.737 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:14:16.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-15T13:14:16.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:16.943 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: get_osds pool-jerasure myobject 2024-05-15T13:14:16.943 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T13:14:16.944 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-05-15T13:14:16.944 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-05-15T13:14:16.944 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T13:14:17.413 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T13:14:17.413 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:14:17.413 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T13:14:17.413 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T13:14:17.413 INFO:tasks.workunit.client.0.smithi005.stderr:2' 2024-05-15T13:14:17.413 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2 2024-05-15T13:14:17.413 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: initial_osds=('3' '1' '4' '0' '2') 2024-05-15T13:14:17.413 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: local -a initial_osds 2024-05-15T13:14:17.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:629: TEST_ec_recovery_unfound: local last_osd=2 2024-05-15T13:14:17.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:630: TEST_ec_recovery_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-05-15T13:14:17.414 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:14:17.414 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:14:17.414 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:14:17.415 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:14:17.415 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:14:18.726 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:14:18.726 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:631: TEST_ec_recovery_unfound: ceph osd down 2 2024-05-15T13:14:19.385 INFO:tasks.workunit.client.0.smithi005.stderr:osd.2 is already down. 2024-05-15T13:14:19.402 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:632: TEST_ec_recovery_unfound: ceph osd out 2 2024-05-15T13:14:19.986 INFO:tasks.workunit.client.0.smithi005.stderr:osd.2 is already out. 2024-05-15T13:14:20.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:634: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-05-15T13:14:20.592 INFO:tasks.workunit.client.0.smithi005.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-05-15T13:14:20.593 INFO:tasks.workunit.client.0.smithi005.stdout:2.0 1 0 1 0 0 4096 0 0 1 0 1 active+undersized+degraded 2024-05-15T13:14:19.225146+0000 45'1 48:26 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2024-05-15T13:14:08.249128+0000 0'0 2024-05-15T13:14:08.249128+0000 0 0 no scrub is scheduled 0 0 2024-05-15T13:14:20.593 INFO:tasks.workunit.client.0.smithi005.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T13:14:07.449836+0000 0'0 45:21 [4,1,2] 4 [4,1,2] 4 0'0 2024-05-15T13:10:49.046418+0000 0'0 2024-05-15T13:10:49.046418+0000 0 0 periodic scrub scheduled @ 2024-05-16T18:04:12.590896+0000 0 0 2024-05-15T13:14:20.593 INFO:tasks.workunit.client.0.smithi005.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T13:14:07.340473+0000 0'0 45:104 [0,4,1] 0 [0,4,1] 0 0'0 2024-05-15T13:10:49.046418+0000 0'0 2024-05-15T13:10:49.046418+0000 0 0 periodic scrub scheduled @ 2024-05-16T23:50:27.372650+0000 0 0 2024-05-15T13:14:20.593 INFO:tasks.workunit.client.0.smithi005.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T13:14:07.254980+0000 0'0 45:21 [4,3,0] 4 [4,3,0] 4 0'0 2024-05-15T13:10:49.046418+0000 0'0 2024-05-15T13:10:49.046418+0000 0 0 periodic scrub scheduled @ 2024-05-16T23:45:47.610880+0000 0 0 2024-05-15T13:14:20.593 INFO:tasks.workunit.client.0.smithi005.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 peering 2024-05-15T13:14:20.162860+0000 0'0 48:81 [1,0,4] 1 [1,0,4] 1 0'0 2024-05-15T13:10:49.046418+0000 0'0 2024-05-15T13:10:49.046418+0000 0 0 no scrub is scheduled 0 0 2024-05-15T13:14:20.593 INFO:tasks.workunit.client.0.smithi005.stdout: 2024-05-15T13:14:20.593 INFO:tasks.workunit.client.0.smithi005.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-05-15T13:14:20.593 INFO:tasks.workunit.client.0.smithi005.stderr:dumped pgs 2024-05-15T13:14:20.610 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:636: TEST_ec_recovery_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2024-05-15T13:14:20.611 INFO:tasks.workunit.client.0.smithi005.stderr:4+0 records in 2024-05-15T13:14:20.611 INFO:tasks.workunit.client.0.smithi005.stderr:4+0 records out 2024-05-15T13:14:20.611 INFO:tasks.workunit.client.0.smithi005.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000165582 s, 24.7 MB/s 2024-05-15T13:14:20.612 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: seq 1 100 2024-05-15T13:14:20.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:20.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:21.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:21.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:21.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:21.424 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:21.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:21.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:21.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:21.994 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:22.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:22.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:22.648 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:22.648 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:22.941 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:22.941 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:23.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:23.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:23.478 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:23.478 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:23.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:23.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:24.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:24.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:24.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:24.337 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:24.596 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:24.596 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:24.898 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:24.898 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:25.158 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:25.158 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:25.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:25.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:25.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:25.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:25.968 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:25.968 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:26.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:26.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:26.574 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:26.574 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:26.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:26.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:27.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:27.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:27.529 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:27.529 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:27.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:27.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:28.166 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:28.166 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:28.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:28.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:28.702 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:28.702 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:28.955 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:28.955 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:29.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:29.274 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:29.567 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:29.567 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:29.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:29.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:30.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:30.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:30.457 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:30.457 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:30.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:30.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:31.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:31.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:31.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:31.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:31.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:31.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:31.870 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:31.870 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:32.166 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:32.166 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:32.551 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:32.551 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:32.777 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:32.777 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:33.089 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:33.089 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:33.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:33.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:33.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:33.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:33.942 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:33.942 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:34.220 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:34.221 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:34.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:34.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:34.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:34.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:35.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:35.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:35.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:35.411 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:35.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:35.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:35.972 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:35.972 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:36.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:36.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:36.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:36.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:36.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:36.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:37.094 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:37.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:37.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:37.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:37.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:37.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:37.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:37.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:38.276 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:38.276 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:38.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:38.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:38.847 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:38.847 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:39.140 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:39.140 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:39.374 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:39.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:39.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:39.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:39.936 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:39.936 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:40.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:40.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:40.583 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:40.583 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:40.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:40.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:41.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:41.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:41.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:41.454 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:41.781 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:41.781 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:42.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:42.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:42.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:42.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:42.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:42.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:42.929 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:42.929 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:43.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:43.214 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:43.524 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:43.524 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:43.792 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:43.792 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:44.096 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:44.096 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:44.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:44.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:44.656 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:44.656 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:44.932 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:44.932 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:45.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:45.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:45.470 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:45.470 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:45.774 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:45.774 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:46.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:46.074 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:46.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:46.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:46.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:46.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:46.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:46.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:47.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:47.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:47.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:47.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:47.808 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:47.808 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:48.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:48.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:48.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:48.388 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:48.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:48.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:48.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:48.957 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:49.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:49.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:49.552 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:14:49.552 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2024-05-15T13:14:49.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:642: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 0 2024-05-15T13:14:49.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-15T13:14:49.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-15T13:14:49.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-15T13:14:49.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-15T13:14:49.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-15T13:14:49.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-15T13:14:49.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj75 2024-05-15T13:14:49.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-15T13:14:49.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-15T13:14:49.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-15T13:14:49.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-05-15T13:14:49.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-15T13:14:49.874 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj75 2024-05-15T13:14:49.874 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T13:14:49.874 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-05-15T13:14:49.874 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj75 2024-05-15T13:14:49.874 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T13:14:50.324 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T13:14:50.325 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:14:50.325 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T13:14:50.325 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T13:14:50.325 INFO:tasks.workunit.client.0.smithi005.stderr:2147483647' 2024-05-15T13:14:50.325 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-05-15T13:14:50.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-05-15T13:14:50.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-15T13:14:50.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-05-15T13:14:50.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-15T13:14:50.326 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-05-15T13:14:50.326 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-15T13:14:50.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-05-15T13:14:50.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-15T13:14:50.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-05-15T13:14:50.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-15T13:14:50.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-15T13:14:50.327 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-15T13:14:50.328 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-05-15T13:14:50.328 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-15T13:14:50.328 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-15T13:14:50.328 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:14:50.328 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:14:50.328 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:14:50.328 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.3.asok 2024-05-15T13:14:50.328 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.80702/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-05-15T13:14:50.442 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-15T13:14:50.442 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-15T13:14:50.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-15T13:14:50.444 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-05-15T13:14:50.444 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-15T13:14:50.444 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-15T13:14:50.444 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:14:50.444 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:14:50.444 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:14:50.444 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.3.asok 2024-05-15T13:14:50.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-15T13:14:50.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-osd.3.asok injectdataerr pool-jerasure obj75 0 2024-05-15T13:14:50.559 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:643: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 1 2024-05-15T13:14:50.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-15T13:14:50.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-15T13:14:50.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-15T13:14:50.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-15T13:14:50.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-15T13:14:50.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-15T13:14:50.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj75 2024-05-15T13:14:50.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-15T13:14:50.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-15T13:14:50.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-15T13:14:50.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-05-15T13:14:50.560 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-15T13:14:50.560 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj75 2024-05-15T13:14:50.560 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T13:14:50.561 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-05-15T13:14:50.561 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T13:14:50.561 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj75 2024-05-15T13:14:51.016 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T13:14:51.016 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:14:51.016 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T13:14:51.016 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T13:14:51.017 INFO:tasks.workunit.client.0.smithi005.stderr:2147483647' 2024-05-15T13:14:51.017 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-05-15T13:14:51.017 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-05-15T13:14:51.017 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-15T13:14:51.017 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-05-15T13:14:51.017 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-15T13:14:51.017 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/1/type 2024-05-15T13:14:51.019 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-15T13:14:51.019 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-05-15T13:14:51.019 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-15T13:14:51.019 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-05-15T13:14:51.019 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-15T13:14:51.019 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-15T13:14:51.019 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-15T13:14:51.020 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-05-15T13:14:51.020 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-05-15T13:14:51.020 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-05-15T13:14:51.020 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:14:51.020 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:14:51.020 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:14:51.020 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.1.asok 2024-05-15T13:14:51.020 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.80702/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-05-15T13:14:51.130 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-15T13:14:51.131 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-15T13:14:51.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-15T13:14:51.132 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-05-15T13:14:51.132 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-05-15T13:14:51.132 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-05-15T13:14:51.133 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:14:51.133 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:14:51.133 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:14:51.133 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.1.asok 2024-05-15T13:14:51.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-15T13:14:51.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-osd.1.asok injectdataerr pool-jerasure obj75 1 2024-05-15T13:14:51.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:645: TEST_ec_recovery_unfound: activate_osd td/test-erasure-eio 2 2024-05-15T13:14:51.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-15T13:14:51.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-15T13:14:51.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-05-15T13:14:51.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-15T13:14:51.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-05-15T13:14:51.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:14:51.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:14:51.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:14:51.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:14:51.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-15T13:14:51.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-15T13:14:51.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-15T13:14:51.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-15T13:14:51.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:14:51.257 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-15T13:14:51.257 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:14:51.258 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:14:51.259 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:14:51.259 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:14:51.259 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:14:51.259 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:14:51.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:14:51.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:14:51.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:14:51.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:14:51.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:14:51.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:14:51.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:14:51.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:14:51.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-15T13:14:51.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-15T13:14:51.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-05-15T13:14:51.260 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.2 2024-05-15T13:14:51.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-05-15T13:14:51.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:14:51.261 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-05-15T13:14:51.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-05-15T13:14:51.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-15T13:14:51.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-15T13:14:51.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-15T13:14:51.292 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:14:51.293+0000 7f2503e447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:14:51.295 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:14:51.297+0000 7f2503e447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:14:51.296 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:14:51.297+0000 7f2503e447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:14:51.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-05-15T13:14:51.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:14:51.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-15T13:14:51.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:14:51.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:14:51.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:14:51.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:14:51.735 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:14:51.736 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:14:51.736 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:14:51.862 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:14:51.865+0000 7f2503e447c0 -1 Falling back to public interface 2024-05-15T13:14:52.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:14:53.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:14:53.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:14:53.185 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:14:53.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:14:53.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:14:53.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:14:53.605 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:14:54.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:14:54.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:14:54.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:14:54.607 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:14:54.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:14:54.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:14:55.033 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:14:55.207 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:14:55.208+0000 7f2503e447c0 -1 osd.2 45 log_to_monitors true 2024-05-15T13:14:56.034 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:14:56.035 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:14:56.035 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:14:56.035 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:14:56.035 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:14:56.036 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:14:56.463 INFO:tasks.workunit.client.0.smithi005.stdout:osd.2 up out weight 0 up_from 53 up_thru 19 down_at 46 last_clean_interval [19,45) [v2:127.0.0.1:6818/3461462082,v1:127.0.0.1:6819/3461462082] [v2:127.0.0.1:6820/3461462082,v1:127.0.0.1:6821/3461462082] exists,up c32adb32-5c52-4405-97ca-aa0f50e78237 2024-05-15T13:14:56.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:14:56.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:14:56.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:14:56.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:646: TEST_ec_recovery_unfound: ceph osd in 2 2024-05-15T13:14:58.023 INFO:tasks.workunit.client.0.smithi005.stderr:osd.2 is already in. 2024-05-15T13:14:58.046 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: sleep 15 2024-05-15T13:15:13.050 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: seq 1 100 2024-05-15T13:15:13.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: for tmp in $(seq 1 100) 2024-05-15T13:15:13.052 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: get_state 2.0 2024-05-15T13:15:13.052 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-15T13:15:13.052 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-15T13:15:13.053 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-15T13:15:13.053 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-15T13:15:13.474 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: state=active+recovery_unfound+degraded 2024-05-15T13:15:13.474 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: echo active+recovery_unfound+degraded 2024-05-15T13:15:13.474 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: grep recovery_unfound 2024-05-15T13:15:13.476 INFO:tasks.workunit.client.0.smithi005.stdout:active+recovery_unfound+degraded 2024-05-15T13:15:13.476 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' 0 = 0 ']' 2024-05-15T13:15:13.476 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:654: TEST_ec_recovery_unfound: break 2024-05-15T13:15:13.476 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:660: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-05-15T13:15:13.857 INFO:tasks.workunit.client.0.smithi005.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-05-15T13:15:13.858 INFO:tasks.workunit.client.0.smithi005.stdout:2.0 101 1 3 0 1 413696 0 0 101 0 101 active+recovery_unfound+degraded 2024-05-15T13:15:02.567494+0000 51'101 57:349 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-05-15T13:14:08.249128+0000 0'0 2024-05-15T13:14:08.249128+0000 0 0 no scrub is scheduled 0 0 2024-05-15T13:15:13.858 INFO:tasks.workunit.client.0.smithi005.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T13:14:59.040920+0000 0'0 57:57 [4,1,2] 4 [4,1,2] 4 0'0 2024-05-15T13:10:49.046418+0000 0'0 2024-05-15T13:10:49.046418+0000 0 0 periodic scrub scheduled @ 2024-05-16T19:45:21.246857+0000 0 0 2024-05-15T13:15:13.858 INFO:tasks.workunit.client.0.smithi005.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T13:14:07.340473+0000 0'0 57:126 [0,4,1] 0 [0,4,1] 0 0'0 2024-05-15T13:10:49.046418+0000 0'0 2024-05-15T13:10:49.046418+0000 0 0 periodic scrub scheduled @ 2024-05-16T23:50:27.372650+0000 0 0 2024-05-15T13:15:13.858 INFO:tasks.workunit.client.0.smithi005.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T13:14:07.254980+0000 0'0 57:43 [4,3,0] 4 [4,3,0] 4 0'0 2024-05-15T13:10:49.046418+0000 0'0 2024-05-15T13:10:49.046418+0000 0 0 periodic scrub scheduled @ 2024-05-16T23:45:47.610880+0000 0 0 2024-05-15T13:15:13.858 INFO:tasks.workunit.client.0.smithi005.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-15T13:14:59.040817+0000 0'0 57:105 [1,0,2] 1 [1,0,2] 1 0'0 2024-05-15T13:10:49.046418+0000 0'0 2024-05-15T13:10:49.046418+0000 0 0 periodic scrub scheduled @ 2024-05-16T16:48:40.660074+0000 0 0 2024-05-15T13:15:13.858 INFO:tasks.workunit.client.0.smithi005.stdout: 2024-05-15T13:15:13.858 INFO:tasks.workunit.client.0.smithi005.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-05-15T13:15:13.858 INFO:tasks.workunit.client.0.smithi005.stderr:dumped pgs 2024-05-15T13:15:13.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:661: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-05-15T13:15:14.057 INFO:tasks.workunit.client.0.smithi005.stdout:{ 2024-05-15T13:15:14.057 INFO:tasks.workunit.client.0.smithi005.stdout: "num_missing": 1, 2024-05-15T13:15:14.057 INFO:tasks.workunit.client.0.smithi005.stdout: "num_unfound": 1, 2024-05-15T13:15:14.057 INFO:tasks.workunit.client.0.smithi005.stdout: "objects": [ 2024-05-15T13:15:14.057 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T13:15:14.057 INFO:tasks.workunit.client.0.smithi005.stdout: "oid": { 2024-05-15T13:15:14.057 INFO:tasks.workunit.client.0.smithi005.stdout: "oid": "obj75", 2024-05-15T13:15:14.057 INFO:tasks.workunit.client.0.smithi005.stdout: "key": "", 2024-05-15T13:15:14.057 INFO:tasks.workunit.client.0.smithi005.stdout: "snapid": -2, 2024-05-15T13:15:14.058 INFO:tasks.workunit.client.0.smithi005.stdout: "hash": 3565005477, 2024-05-15T13:15:14.058 INFO:tasks.workunit.client.0.smithi005.stdout: "max": 0, 2024-05-15T13:15:14.058 INFO:tasks.workunit.client.0.smithi005.stdout: "pool": 2, 2024-05-15T13:15:14.058 INFO:tasks.workunit.client.0.smithi005.stdout: "namespace": "" 2024-05-15T13:15:14.058 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.058 INFO:tasks.workunit.client.0.smithi005.stdout: "need": "51'76", 2024-05-15T13:15:14.058 INFO:tasks.workunit.client.0.smithi005.stdout: "have": "0'0", 2024-05-15T13:15:14.058 INFO:tasks.workunit.client.0.smithi005.stdout: "flags": "none", 2024-05-15T13:15:14.058 INFO:tasks.workunit.client.0.smithi005.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-05-15T13:15:14.058 INFO:tasks.workunit.client.0.smithi005.stdout: "locations": [ 2024-05-15T13:15:14.058 INFO:tasks.workunit.client.0.smithi005.stdout: "0(3)", 2024-05-15T13:15:14.058 INFO:tasks.workunit.client.0.smithi005.stdout: "4(2)" 2024-05-15T13:15:14.058 INFO:tasks.workunit.client.0.smithi005.stdout: ] 2024-05-15T13:15:14.059 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T13:15:14.059 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.059 INFO:tasks.workunit.client.0.smithi005.stdout: "state": "NotRecovering", 2024-05-15T13:15:14.059 INFO:tasks.workunit.client.0.smithi005.stdout: "available_might_have_unfound": true, 2024-05-15T13:15:14.059 INFO:tasks.workunit.client.0.smithi005.stdout: "might_have_unfound": [], 2024-05-15T13:15:14.059 INFO:tasks.workunit.client.0.smithi005.stdout: "more": false 2024-05-15T13:15:14.059 INFO:tasks.workunit.client.0.smithi005.stdout:} 2024-05-15T13:15:14.071 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:662: TEST_ec_recovery_unfound: ceph pg 2.0 query 2024-05-15T13:15:14.219 INFO:tasks.workunit.client.0.smithi005.stdout:{ 2024-05-15T13:15:14.220 INFO:tasks.workunit.client.0.smithi005.stdout: "snap_trimq": "[]", 2024-05-15T13:15:14.220 INFO:tasks.workunit.client.0.smithi005.stdout: "snap_trimq_len": 0, 2024-05-15T13:15:14.220 INFO:tasks.workunit.client.0.smithi005.stdout: "state": "active+recovery_unfound+degraded", 2024-05-15T13:15:14.220 INFO:tasks.workunit.client.0.smithi005.stdout: "epoch": 57, 2024-05-15T13:15:14.220 INFO:tasks.workunit.client.0.smithi005.stdout: "up": [ 2024-05-15T13:15:14.220 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T13:15:14.220 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T13:15:14.220 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T13:15:14.220 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T13:15:14.220 INFO:tasks.workunit.client.0.smithi005.stdout: 2 2024-05-15T13:15:14.220 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.220 INFO:tasks.workunit.client.0.smithi005.stdout: "acting": [ 2024-05-15T13:15:14.220 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T13:15:14.220 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T13:15:14.221 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T13:15:14.221 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T13:15:14.221 INFO:tasks.workunit.client.0.smithi005.stdout: 2 2024-05-15T13:15:14.221 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.221 INFO:tasks.workunit.client.0.smithi005.stdout: "acting_recovery_backfill": [ 2024-05-15T13:15:14.221 INFO:tasks.workunit.client.0.smithi005.stdout: "0(3)", 2024-05-15T13:15:14.221 INFO:tasks.workunit.client.0.smithi005.stdout: "1(1)", 2024-05-15T13:15:14.221 INFO:tasks.workunit.client.0.smithi005.stdout: "2(4)", 2024-05-15T13:15:14.221 INFO:tasks.workunit.client.0.smithi005.stdout: "3(0)", 2024-05-15T13:15:14.221 INFO:tasks.workunit.client.0.smithi005.stdout: "4(2)" 2024-05-15T13:15:14.221 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.221 INFO:tasks.workunit.client.0.smithi005.stdout: "info": { 2024-05-15T13:15:14.221 INFO:tasks.workunit.client.0.smithi005.stdout: "pgid": "2.0s0", 2024-05-15T13:15:14.221 INFO:tasks.workunit.client.0.smithi005.stdout: "shared": "0", 2024-05-15T13:15:14.221 INFO:tasks.workunit.client.0.smithi005.stdout: "last_update": "51'101", 2024-05-15T13:15:14.222 INFO:tasks.workunit.client.0.smithi005.stdout: "last_complete": "51'75", 2024-05-15T13:15:14.222 INFO:tasks.workunit.client.0.smithi005.stdout: "log_tail": "0'0", 2024-05-15T13:15:14.222 INFO:tasks.workunit.client.0.smithi005.stdout: "last_user_version": 101, 2024-05-15T13:15:14.222 INFO:tasks.workunit.client.0.smithi005.stdout: "last_backfill": "MAX", 2024-05-15T13:15:14.222 INFO:tasks.workunit.client.0.smithi005.stdout: "purged_snaps": [], 2024-05-15T13:15:14.222 INFO:tasks.workunit.client.0.smithi005.stdout: "history": { 2024-05-15T13:15:14.222 INFO:tasks.workunit.client.0.smithi005.stdout: "epoch_created": 43, 2024-05-15T13:15:14.222 INFO:tasks.workunit.client.0.smithi005.stdout: "epoch_pool_created": 43, 2024-05-15T13:15:14.222 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_started": 57, 2024-05-15T13:15:14.222 INFO:tasks.workunit.client.0.smithi005.stdout: "last_interval_started": 55, 2024-05-15T13:15:14.222 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_clean": 44, 2024-05-15T13:15:14.222 INFO:tasks.workunit.client.0.smithi005.stdout: "last_interval_clean": 43, 2024-05-15T13:15:14.222 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_split": 0, 2024-05-15T13:15:14.223 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_marked_full": 0, 2024-05-15T13:15:14.223 INFO:tasks.workunit.client.0.smithi005.stdout: "same_up_since": 55, 2024-05-15T13:15:14.223 INFO:tasks.workunit.client.0.smithi005.stdout: "same_interval_since": 55, 2024-05-15T13:15:14.223 INFO:tasks.workunit.client.0.smithi005.stdout: "same_primary_since": 43, 2024-05-15T13:15:14.223 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub": "0'0", 2024-05-15T13:15:14.223 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.223 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub": "0'0", 2024-05-15T13:15:14.223 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.223 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.223 INFO:tasks.workunit.client.0.smithi005.stdout: "prior_readable_until_ub": 0 2024-05-15T13:15:14.223 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.223 INFO:tasks.workunit.client.0.smithi005.stdout: "stats": { 2024-05-15T13:15:14.223 INFO:tasks.workunit.client.0.smithi005.stdout: "version": "51'101", 2024-05-15T13:15:14.223 INFO:tasks.workunit.client.0.smithi005.stdout: "reported_seq": 350, 2024-05-15T13:15:14.224 INFO:tasks.workunit.client.0.smithi005.stdout: "reported_epoch": 57, 2024-05-15T13:15:14.224 INFO:tasks.workunit.client.0.smithi005.stdout: "state": "active+recovery_unfound+degraded", 2024-05-15T13:15:14.224 INFO:tasks.workunit.client.0.smithi005.stdout: "last_fresh": "2024-05-15T13:15:02.607133+0000", 2024-05-15T13:15:14.224 INFO:tasks.workunit.client.0.smithi005.stdout: "last_change": "2024-05-15T13:15:02.567494+0000", 2024-05-15T13:15:14.224 INFO:tasks.workunit.client.0.smithi005.stdout: "last_active": "2024-05-15T13:15:02.607133+0000", 2024-05-15T13:15:14.224 INFO:tasks.workunit.client.0.smithi005.stdout: "last_peered": "2024-05-15T13:15:02.607133+0000", 2024-05-15T13:15:14.224 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean": "2024-05-15T13:14:16.946278+0000", 2024-05-15T13:15:14.224 INFO:tasks.workunit.client.0.smithi005.stdout: "last_became_active": "2024-05-15T13:14:59.083619+0000", 2024-05-15T13:15:14.224 INFO:tasks.workunit.client.0.smithi005.stdout: "last_became_peered": "2024-05-15T13:14:59.083619+0000", 2024-05-15T13:15:14.224 INFO:tasks.workunit.client.0.smithi005.stdout: "last_unstale": "2024-05-15T13:15:02.607133+0000", 2024-05-15T13:15:14.224 INFO:tasks.workunit.client.0.smithi005.stdout: "last_undegraded": "2024-05-15T13:14:58.643132+0000", 2024-05-15T13:15:14.224 INFO:tasks.workunit.client.0.smithi005.stdout: "last_fullsized": "2024-05-15T13:15:02.607133+0000", 2024-05-15T13:15:14.224 INFO:tasks.workunit.client.0.smithi005.stdout: "mapping_epoch": 55, 2024-05-15T13:15:14.224 INFO:tasks.workunit.client.0.smithi005.stdout: "log_start": "0'0", 2024-05-15T13:15:14.224 INFO:tasks.workunit.client.0.smithi005.stdout: "ondisk_log_start": "0'0", 2024-05-15T13:15:14.224 INFO:tasks.workunit.client.0.smithi005.stdout: "created": 43, 2024-05-15T13:15:14.225 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_clean": 44, 2024-05-15T13:15:14.225 INFO:tasks.workunit.client.0.smithi005.stdout: "parent": "0.0", 2024-05-15T13:15:14.225 INFO:tasks.workunit.client.0.smithi005.stdout: "parent_split_bits": 0, 2024-05-15T13:15:14.225 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub": "0'0", 2024-05-15T13:15:14.225 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.225 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub": "0'0", 2024-05-15T13:15:14.225 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.225 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.225 INFO:tasks.workunit.client.0.smithi005.stdout: "objects_scrubbed": 0, 2024-05-15T13:15:14.225 INFO:tasks.workunit.client.0.smithi005.stdout: "log_size": 101, 2024-05-15T13:15:14.225 INFO:tasks.workunit.client.0.smithi005.stdout: "log_dups_size": 0, 2024-05-15T13:15:14.225 INFO:tasks.workunit.client.0.smithi005.stdout: "ondisk_log_size": 101, 2024-05-15T13:15:14.225 INFO:tasks.workunit.client.0.smithi005.stdout: "stats_invalid": false, 2024-05-15T13:15:14.225 INFO:tasks.workunit.client.0.smithi005.stdout: "dirty_stats_invalid": false, 2024-05-15T13:15:14.225 INFO:tasks.workunit.client.0.smithi005.stdout: "omap_stats_invalid": false, 2024-05-15T13:15:14.226 INFO:tasks.workunit.client.0.smithi005.stdout: "hitset_stats_invalid": false, 2024-05-15T13:15:14.226 INFO:tasks.workunit.client.0.smithi005.stdout: "hitset_bytes_stats_invalid": false, 2024-05-15T13:15:14.226 INFO:tasks.workunit.client.0.smithi005.stdout: "pin_stats_invalid": false, 2024-05-15T13:15:14.226 INFO:tasks.workunit.client.0.smithi005.stdout: "manifest_stats_invalid": false, 2024-05-15T13:15:14.226 INFO:tasks.workunit.client.0.smithi005.stdout: "snaptrimq_len": 0, 2024-05-15T13:15:14.226 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_duration": 0, 2024-05-15T13:15:14.226 INFO:tasks.workunit.client.0.smithi005.stdout: "scrub_schedule": "no scrub is scheduled", 2024-05-15T13:15:14.226 INFO:tasks.workunit.client.0.smithi005.stdout: "scrub_duration": 0, 2024-05-15T13:15:14.226 INFO:tasks.workunit.client.0.smithi005.stdout: "objects_trimmed": 0, 2024-05-15T13:15:14.226 INFO:tasks.workunit.client.0.smithi005.stdout: "snaptrim_duration": 0, 2024-05-15T13:15:14.226 INFO:tasks.workunit.client.0.smithi005.stdout: "stat_sum": { 2024-05-15T13:15:14.226 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes": 413696, 2024-05-15T13:15:14.226 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects": 101, 2024-05-15T13:15:14.227 INFO:tasks.workunit.client.0.smithi005.stdout: "num_object_clones": 0, 2024-05-15T13:15:14.227 INFO:tasks.workunit.client.0.smithi005.stdout: "num_object_copies": 505, 2024-05-15T13:15:14.227 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_missing_on_primary": 1, 2024-05-15T13:15:14.227 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_missing": 0, 2024-05-15T13:15:14.227 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_degraded": 3, 2024-05-15T13:15:14.227 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_misplaced": 0, 2024-05-15T13:15:14.227 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_unfound": 1, 2024-05-15T13:15:14.227 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_dirty": 101, 2024-05-15T13:15:14.227 INFO:tasks.workunit.client.0.smithi005.stdout: "num_whiteouts": 0, 2024-05-15T13:15:14.227 INFO:tasks.workunit.client.0.smithi005.stdout: "num_read": 0, 2024-05-15T13:15:14.227 INFO:tasks.workunit.client.0.smithi005.stdout: "num_read_kb": 0, 2024-05-15T13:15:14.227 INFO:tasks.workunit.client.0.smithi005.stdout: "num_write": 101, 2024-05-15T13:15:14.227 INFO:tasks.workunit.client.0.smithi005.stdout: "num_write_kb": 404, 2024-05-15T13:15:14.227 INFO:tasks.workunit.client.0.smithi005.stdout: "num_scrub_errors": 0, 2024-05-15T13:15:14.228 INFO:tasks.workunit.client.0.smithi005.stdout: "num_shallow_scrub_errors": 0, 2024-05-15T13:15:14.228 INFO:tasks.workunit.client.0.smithi005.stdout: "num_deep_scrub_errors": 0, 2024-05-15T13:15:14.228 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_recovered": 99, 2024-05-15T13:15:14.228 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes_recovered": 405504, 2024-05-15T13:15:14.228 INFO:tasks.workunit.client.0.smithi005.stdout: "num_keys_recovered": 0, 2024-05-15T13:15:14.228 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_omap": 0, 2024-05-15T13:15:14.228 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_hit_set_archive": 0, 2024-05-15T13:15:14.228 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes_hit_set_archive": 0, 2024-05-15T13:15:14.228 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush": 0, 2024-05-15T13:15:14.228 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_kb": 0, 2024-05-15T13:15:14.228 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict": 0, 2024-05-15T13:15:14.228 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_kb": 0, 2024-05-15T13:15:14.228 INFO:tasks.workunit.client.0.smithi005.stdout: "num_promote": 0, 2024-05-15T13:15:14.228 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_mode_high": 0, 2024-05-15T13:15:14.229 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_mode_low": 0, 2024-05-15T13:15:14.229 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_mode_some": 0, 2024-05-15T13:15:14.229 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_mode_full": 0, 2024-05-15T13:15:14.229 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_pinned": 0, 2024-05-15T13:15:14.229 INFO:tasks.workunit.client.0.smithi005.stdout: "num_legacy_snapsets": 0, 2024-05-15T13:15:14.229 INFO:tasks.workunit.client.0.smithi005.stdout: "num_large_omap_objects": 0, 2024-05-15T13:15:14.229 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_manifest": 0, 2024-05-15T13:15:14.229 INFO:tasks.workunit.client.0.smithi005.stdout: "num_omap_bytes": 0, 2024-05-15T13:15:14.229 INFO:tasks.workunit.client.0.smithi005.stdout: "num_omap_keys": 0, 2024-05-15T13:15:14.230 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_repaired": 0 2024-05-15T13:15:14.230 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.230 INFO:tasks.workunit.client.0.smithi005.stdout: "up": [ 2024-05-15T13:15:14.230 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T13:15:14.230 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T13:15:14.230 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T13:15:14.230 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T13:15:14.230 INFO:tasks.workunit.client.0.smithi005.stdout: 2 2024-05-15T13:15:14.230 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.230 INFO:tasks.workunit.client.0.smithi005.stdout: "acting": [ 2024-05-15T13:15:14.230 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T13:15:14.230 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T13:15:14.230 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T13:15:14.230 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T13:15:14.231 INFO:tasks.workunit.client.0.smithi005.stdout: 2 2024-05-15T13:15:14.231 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.231 INFO:tasks.workunit.client.0.smithi005.stdout: "avail_no_missing": [ 2024-05-15T13:15:14.231 INFO:tasks.workunit.client.0.smithi005.stdout: "0(3)", 2024-05-15T13:15:14.231 INFO:tasks.workunit.client.0.smithi005.stdout: "4(2)" 2024-05-15T13:15:14.231 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.231 INFO:tasks.workunit.client.0.smithi005.stdout: "object_location_counts": [ 2024-05-15T13:15:14.231 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T13:15:14.231 INFO:tasks.workunit.client.0.smithi005.stdout: "shards": "0(3),1(1),2(4),3(0),4(2)", 2024-05-15T13:15:14.231 INFO:tasks.workunit.client.0.smithi005.stdout: "objects": 100 2024-05-15T13:15:14.231 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.231 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T13:15:14.231 INFO:tasks.workunit.client.0.smithi005.stdout: "shards": "0(3),4(2)", 2024-05-15T13:15:14.231 INFO:tasks.workunit.client.0.smithi005.stdout: "objects": 1 2024-05-15T13:15:14.231 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T13:15:14.232 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.232 INFO:tasks.workunit.client.0.smithi005.stdout: "blocked_by": [], 2024-05-15T13:15:14.232 INFO:tasks.workunit.client.0.smithi005.stdout: "up_primary": 3, 2024-05-15T13:15:14.232 INFO:tasks.workunit.client.0.smithi005.stdout: "acting_primary": 3, 2024-05-15T13:15:14.232 INFO:tasks.workunit.client.0.smithi005.stdout: "purged_snaps": [] 2024-05-15T13:15:14.232 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.232 INFO:tasks.workunit.client.0.smithi005.stdout: "empty": 0, 2024-05-15T13:15:14.232 INFO:tasks.workunit.client.0.smithi005.stdout: "dne": 0, 2024-05-15T13:15:14.232 INFO:tasks.workunit.client.0.smithi005.stdout: "incomplete": 0, 2024-05-15T13:15:14.232 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_started": 57, 2024-05-15T13:15:14.232 INFO:tasks.workunit.client.0.smithi005.stdout: "hit_set_history": { 2024-05-15T13:15:14.232 INFO:tasks.workunit.client.0.smithi005.stdout: "current_last_update": "0'0", 2024-05-15T13:15:14.232 INFO:tasks.workunit.client.0.smithi005.stdout: "history": [] 2024-05-15T13:15:14.233 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T13:15:14.233 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.233 INFO:tasks.workunit.client.0.smithi005.stdout: "peer_info": [ 2024-05-15T13:15:14.233 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T13:15:14.233 INFO:tasks.workunit.client.0.smithi005.stdout: "peer": "0(3)", 2024-05-15T13:15:14.233 INFO:tasks.workunit.client.0.smithi005.stdout: "pgid": "2.0s3", 2024-05-15T13:15:14.233 INFO:tasks.workunit.client.0.smithi005.stdout: "shared": "3", 2024-05-15T13:15:14.234 INFO:tasks.workunit.client.0.smithi005.stdout: "last_update": "51'101", 2024-05-15T13:15:14.234 INFO:tasks.workunit.client.0.smithi005.stdout: "last_complete": "51'101", 2024-05-15T13:15:14.234 INFO:tasks.workunit.client.0.smithi005.stdout: "log_tail": "0'0", 2024-05-15T13:15:14.234 INFO:tasks.workunit.client.0.smithi005.stdout: "last_user_version": 101, 2024-05-15T13:15:14.234 INFO:tasks.workunit.client.0.smithi005.stdout: "last_backfill": "MAX", 2024-05-15T13:15:14.234 INFO:tasks.workunit.client.0.smithi005.stdout: "purged_snaps": [], 2024-05-15T13:15:14.235 INFO:tasks.workunit.client.0.smithi005.stdout: "history": { 2024-05-15T13:15:14.235 INFO:tasks.workunit.client.0.smithi005.stdout: "epoch_created": 43, 2024-05-15T13:15:14.235 INFO:tasks.workunit.client.0.smithi005.stdout: "epoch_pool_created": 43, 2024-05-15T13:15:14.235 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_started": 57, 2024-05-15T13:15:14.235 INFO:tasks.workunit.client.0.smithi005.stdout: "last_interval_started": 55, 2024-05-15T13:15:14.235 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_clean": 44, 2024-05-15T13:15:14.235 INFO:tasks.workunit.client.0.smithi005.stdout: "last_interval_clean": 43, 2024-05-15T13:15:14.235 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_split": 0, 2024-05-15T13:15:14.235 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_marked_full": 0, 2024-05-15T13:15:14.235 INFO:tasks.workunit.client.0.smithi005.stdout: "same_up_since": 55, 2024-05-15T13:15:14.235 INFO:tasks.workunit.client.0.smithi005.stdout: "same_interval_since": 55, 2024-05-15T13:15:14.235 INFO:tasks.workunit.client.0.smithi005.stdout: "same_primary_since": 43, 2024-05-15T13:15:14.235 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub": "0'0", 2024-05-15T13:15:14.236 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.236 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub": "0'0", 2024-05-15T13:15:14.236 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.236 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.236 INFO:tasks.workunit.client.0.smithi005.stdout: "prior_readable_until_ub": 0 2024-05-15T13:15:14.236 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.236 INFO:tasks.workunit.client.0.smithi005.stdout: "stats": { 2024-05-15T13:15:14.236 INFO:tasks.workunit.client.0.smithi005.stdout: "version": "51'101", 2024-05-15T13:15:14.236 INFO:tasks.workunit.client.0.smithi005.stdout: "reported_seq": 132, 2024-05-15T13:15:14.236 INFO:tasks.workunit.client.0.smithi005.stdout: "reported_epoch": 51, 2024-05-15T13:15:14.236 INFO:tasks.workunit.client.0.smithi005.stdout: "state": "active+undersized+degraded", 2024-05-15T13:15:14.236 INFO:tasks.workunit.client.0.smithi005.stdout: "last_fresh": "2024-05-15T13:14:49.873721+0000", 2024-05-15T13:15:14.236 INFO:tasks.workunit.client.0.smithi005.stdout: "last_change": "2024-05-15T13:14:19.225146+0000", 2024-05-15T13:15:14.236 INFO:tasks.workunit.client.0.smithi005.stdout: "last_active": "2024-05-15T13:14:49.873721+0000", 2024-05-15T13:15:14.236 INFO:tasks.workunit.client.0.smithi005.stdout: "last_peered": "2024-05-15T13:14:49.873721+0000", 2024-05-15T13:15:14.237 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean": "2024-05-15T13:14:16.946278+0000", 2024-05-15T13:15:14.237 INFO:tasks.workunit.client.0.smithi005.stdout: "last_became_active": "2024-05-15T13:14:19.225146+0000", 2024-05-15T13:15:14.237 INFO:tasks.workunit.client.0.smithi005.stdout: "last_became_peered": "2024-05-15T13:14:19.225146+0000", 2024-05-15T13:15:14.237 INFO:tasks.workunit.client.0.smithi005.stdout: "last_unstale": "2024-05-15T13:14:49.873721+0000", 2024-05-15T13:15:14.237 INFO:tasks.workunit.client.0.smithi005.stdout: "last_undegraded": "2024-05-15T13:14:18.809096+0000", 2024-05-15T13:15:14.237 INFO:tasks.workunit.client.0.smithi005.stdout: "last_fullsized": "2024-05-15T13:14:18.808870+0000", 2024-05-15T13:15:14.237 INFO:tasks.workunit.client.0.smithi005.stdout: "mapping_epoch": 55, 2024-05-15T13:15:14.237 INFO:tasks.workunit.client.0.smithi005.stdout: "log_start": "0'0", 2024-05-15T13:15:14.237 INFO:tasks.workunit.client.0.smithi005.stdout: "ondisk_log_start": "0'0", 2024-05-15T13:15:14.237 INFO:tasks.workunit.client.0.smithi005.stdout: "created": 43, 2024-05-15T13:15:14.237 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_clean": 44, 2024-05-15T13:15:14.238 INFO:tasks.workunit.client.0.smithi005.stdout: "parent": "0.0", 2024-05-15T13:15:14.238 INFO:tasks.workunit.client.0.smithi005.stdout: "parent_split_bits": 0, 2024-05-15T13:15:14.238 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub": "0'0", 2024-05-15T13:15:14.238 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.238 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub": "0'0", 2024-05-15T13:15:14.238 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.238 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.238 INFO:tasks.workunit.client.0.smithi005.stdout: "objects_scrubbed": 0, 2024-05-15T13:15:14.238 INFO:tasks.workunit.client.0.smithi005.stdout: "log_size": 101, 2024-05-15T13:15:14.238 INFO:tasks.workunit.client.0.smithi005.stdout: "log_dups_size": 0, 2024-05-15T13:15:14.238 INFO:tasks.workunit.client.0.smithi005.stdout: "ondisk_log_size": 101, 2024-05-15T13:15:14.238 INFO:tasks.workunit.client.0.smithi005.stdout: "stats_invalid": false, 2024-05-15T13:15:14.238 INFO:tasks.workunit.client.0.smithi005.stdout: "dirty_stats_invalid": false, 2024-05-15T13:15:14.238 INFO:tasks.workunit.client.0.smithi005.stdout: "omap_stats_invalid": false, 2024-05-15T13:15:14.239 INFO:tasks.workunit.client.0.smithi005.stdout: "hitset_stats_invalid": false, 2024-05-15T13:15:14.239 INFO:tasks.workunit.client.0.smithi005.stdout: "hitset_bytes_stats_invalid": false, 2024-05-15T13:15:14.239 INFO:tasks.workunit.client.0.smithi005.stdout: "pin_stats_invalid": false, 2024-05-15T13:15:14.239 INFO:tasks.workunit.client.0.smithi005.stdout: "manifest_stats_invalid": false, 2024-05-15T13:15:14.239 INFO:tasks.workunit.client.0.smithi005.stdout: "snaptrimq_len": 0, 2024-05-15T13:15:14.239 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_duration": 0, 2024-05-15T13:15:14.239 INFO:tasks.workunit.client.0.smithi005.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-05-16T19:15:16.210930+0000", 2024-05-15T13:15:14.239 INFO:tasks.workunit.client.0.smithi005.stdout: "scrub_duration": 0, 2024-05-15T13:15:14.239 INFO:tasks.workunit.client.0.smithi005.stdout: "objects_trimmed": 0, 2024-05-15T13:15:14.239 INFO:tasks.workunit.client.0.smithi005.stdout: "snaptrim_duration": 0, 2024-05-15T13:15:14.239 INFO:tasks.workunit.client.0.smithi005.stdout: "stat_sum": { 2024-05-15T13:15:14.239 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes": 413696, 2024-05-15T13:15:14.239 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects": 101, 2024-05-15T13:15:14.240 INFO:tasks.workunit.client.0.smithi005.stdout: "num_object_clones": 0, 2024-05-15T13:15:14.240 INFO:tasks.workunit.client.0.smithi005.stdout: "num_object_copies": 505, 2024-05-15T13:15:14.240 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_missing_on_primary": 0, 2024-05-15T13:15:14.240 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_missing": 0, 2024-05-15T13:15:14.240 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_degraded": 101, 2024-05-15T13:15:14.240 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_misplaced": 0, 2024-05-15T13:15:14.240 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_unfound": 0, 2024-05-15T13:15:14.240 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_dirty": 101, 2024-05-15T13:15:14.240 INFO:tasks.workunit.client.0.smithi005.stdout: "num_whiteouts": 0, 2024-05-15T13:15:14.240 INFO:tasks.workunit.client.0.smithi005.stdout: "num_read": 0, 2024-05-15T13:15:14.240 INFO:tasks.workunit.client.0.smithi005.stdout: "num_read_kb": 0, 2024-05-15T13:15:14.241 INFO:tasks.workunit.client.0.smithi005.stdout: "num_write": 101, 2024-05-15T13:15:14.241 INFO:tasks.workunit.client.0.smithi005.stdout: "num_write_kb": 404, 2024-05-15T13:15:14.241 INFO:tasks.workunit.client.0.smithi005.stdout: "num_scrub_errors": 0, 2024-05-15T13:15:14.241 INFO:tasks.workunit.client.0.smithi005.stdout: "num_shallow_scrub_errors": 0, 2024-05-15T13:15:14.241 INFO:tasks.workunit.client.0.smithi005.stdout: "num_deep_scrub_errors": 0, 2024-05-15T13:15:14.241 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_recovered": 0, 2024-05-15T13:15:14.241 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes_recovered": 0, 2024-05-15T13:15:14.241 INFO:tasks.workunit.client.0.smithi005.stdout: "num_keys_recovered": 0, 2024-05-15T13:15:14.241 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_omap": 0, 2024-05-15T13:15:14.241 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_hit_set_archive": 0, 2024-05-15T13:15:14.241 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes_hit_set_archive": 0, 2024-05-15T13:15:14.241 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush": 0, 2024-05-15T13:15:14.241 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_kb": 0, 2024-05-15T13:15:14.241 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict": 0, 2024-05-15T13:15:14.241 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_kb": 0, 2024-05-15T13:15:14.242 INFO:tasks.workunit.client.0.smithi005.stdout: "num_promote": 0, 2024-05-15T13:15:14.242 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_mode_high": 0, 2024-05-15T13:15:14.242 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_mode_low": 0, 2024-05-15T13:15:14.242 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_mode_some": 0, 2024-05-15T13:15:14.242 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_mode_full": 0, 2024-05-15T13:15:14.242 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_pinned": 0, 2024-05-15T13:15:14.242 INFO:tasks.workunit.client.0.smithi005.stdout: "num_legacy_snapsets": 0, 2024-05-15T13:15:14.242 INFO:tasks.workunit.client.0.smithi005.stdout: "num_large_omap_objects": 0, 2024-05-15T13:15:14.242 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_manifest": 0, 2024-05-15T13:15:14.242 INFO:tasks.workunit.client.0.smithi005.stdout: "num_omap_bytes": 0, 2024-05-15T13:15:14.242 INFO:tasks.workunit.client.0.smithi005.stdout: "num_omap_keys": 0, 2024-05-15T13:15:14.242 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_repaired": 0 2024-05-15T13:15:14.242 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.242 INFO:tasks.workunit.client.0.smithi005.stdout: "up": [ 2024-05-15T13:15:14.243 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T13:15:14.243 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T13:15:14.243 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T13:15:14.243 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T13:15:14.243 INFO:tasks.workunit.client.0.smithi005.stdout: 2 2024-05-15T13:15:14.243 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.243 INFO:tasks.workunit.client.0.smithi005.stdout: "acting": [ 2024-05-15T13:15:14.243 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T13:15:14.243 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T13:15:14.243 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T13:15:14.243 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T13:15:14.243 INFO:tasks.workunit.client.0.smithi005.stdout: 2 2024-05-15T13:15:14.243 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.243 INFO:tasks.workunit.client.0.smithi005.stdout: "avail_no_missing": [ 2024-05-15T13:15:14.243 INFO:tasks.workunit.client.0.smithi005.stdout: "3(0)", 2024-05-15T13:15:14.244 INFO:tasks.workunit.client.0.smithi005.stdout: "0(3)", 2024-05-15T13:15:14.244 INFO:tasks.workunit.client.0.smithi005.stdout: "1(1)", 2024-05-15T13:15:14.244 INFO:tasks.workunit.client.0.smithi005.stdout: "4(2)" 2024-05-15T13:15:14.244 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.244 INFO:tasks.workunit.client.0.smithi005.stdout: "object_location_counts": [ 2024-05-15T13:15:14.244 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T13:15:14.244 INFO:tasks.workunit.client.0.smithi005.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-05-15T13:15:14.244 INFO:tasks.workunit.client.0.smithi005.stdout: "objects": 101 2024-05-15T13:15:14.244 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T13:15:14.244 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.244 INFO:tasks.workunit.client.0.smithi005.stdout: "blocked_by": [], 2024-05-15T13:15:14.244 INFO:tasks.workunit.client.0.smithi005.stdout: "up_primary": 3, 2024-05-15T13:15:14.244 INFO:tasks.workunit.client.0.smithi005.stdout: "acting_primary": 3, 2024-05-15T13:15:14.245 INFO:tasks.workunit.client.0.smithi005.stdout: "purged_snaps": [] 2024-05-15T13:15:14.245 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.245 INFO:tasks.workunit.client.0.smithi005.stdout: "empty": 0, 2024-05-15T13:15:14.245 INFO:tasks.workunit.client.0.smithi005.stdout: "dne": 0, 2024-05-15T13:15:14.245 INFO:tasks.workunit.client.0.smithi005.stdout: "incomplete": 0, 2024-05-15T13:15:14.245 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_started": 57, 2024-05-15T13:15:14.245 INFO:tasks.workunit.client.0.smithi005.stdout: "hit_set_history": { 2024-05-15T13:15:14.245 INFO:tasks.workunit.client.0.smithi005.stdout: "current_last_update": "0'0", 2024-05-15T13:15:14.245 INFO:tasks.workunit.client.0.smithi005.stdout: "history": [] 2024-05-15T13:15:14.245 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T13:15:14.245 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.245 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T13:15:14.245 INFO:tasks.workunit.client.0.smithi005.stdout: "peer": "1(1)", 2024-05-15T13:15:14.245 INFO:tasks.workunit.client.0.smithi005.stdout: "pgid": "2.0s1", 2024-05-15T13:15:14.245 INFO:tasks.workunit.client.0.smithi005.stdout: "shared": "1", 2024-05-15T13:15:14.245 INFO:tasks.workunit.client.0.smithi005.stdout: "last_update": "51'101", 2024-05-15T13:15:14.246 INFO:tasks.workunit.client.0.smithi005.stdout: "last_complete": "51'101", 2024-05-15T13:15:14.246 INFO:tasks.workunit.client.0.smithi005.stdout: "log_tail": "0'0", 2024-05-15T13:15:14.246 INFO:tasks.workunit.client.0.smithi005.stdout: "last_user_version": 101, 2024-05-15T13:15:14.246 INFO:tasks.workunit.client.0.smithi005.stdout: "last_backfill": "MAX", 2024-05-15T13:15:14.246 INFO:tasks.workunit.client.0.smithi005.stdout: "purged_snaps": [], 2024-05-15T13:15:14.246 INFO:tasks.workunit.client.0.smithi005.stdout: "history": { 2024-05-15T13:15:14.246 INFO:tasks.workunit.client.0.smithi005.stdout: "epoch_created": 43, 2024-05-15T13:15:14.246 INFO:tasks.workunit.client.0.smithi005.stdout: "epoch_pool_created": 43, 2024-05-15T13:15:14.246 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_started": 57, 2024-05-15T13:15:14.246 INFO:tasks.workunit.client.0.smithi005.stdout: "last_interval_started": 55, 2024-05-15T13:15:14.246 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_clean": 44, 2024-05-15T13:15:14.246 INFO:tasks.workunit.client.0.smithi005.stdout: "last_interval_clean": 43, 2024-05-15T13:15:14.246 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_split": 0, 2024-05-15T13:15:14.246 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_marked_full": 0, 2024-05-15T13:15:14.247 INFO:tasks.workunit.client.0.smithi005.stdout: "same_up_since": 55, 2024-05-15T13:15:14.247 INFO:tasks.workunit.client.0.smithi005.stdout: "same_interval_since": 55, 2024-05-15T13:15:14.247 INFO:tasks.workunit.client.0.smithi005.stdout: "same_primary_since": 43, 2024-05-15T13:15:14.247 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub": "0'0", 2024-05-15T13:15:14.247 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.247 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub": "0'0", 2024-05-15T13:15:14.247 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.247 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.247 INFO:tasks.workunit.client.0.smithi005.stdout: "prior_readable_until_ub": 0 2024-05-15T13:15:14.247 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.247 INFO:tasks.workunit.client.0.smithi005.stdout: "stats": { 2024-05-15T13:15:14.247 INFO:tasks.workunit.client.0.smithi005.stdout: "version": "51'101", 2024-05-15T13:15:14.247 INFO:tasks.workunit.client.0.smithi005.stdout: "reported_seq": 132, 2024-05-15T13:15:14.247 INFO:tasks.workunit.client.0.smithi005.stdout: "reported_epoch": 51, 2024-05-15T13:15:14.247 INFO:tasks.workunit.client.0.smithi005.stdout: "state": "active+undersized+degraded", 2024-05-15T13:15:14.247 INFO:tasks.workunit.client.0.smithi005.stdout: "last_fresh": "2024-05-15T13:14:49.873721+0000", 2024-05-15T13:15:14.247 INFO:tasks.workunit.client.0.smithi005.stdout: "last_change": "2024-05-15T13:14:19.225146+0000", 2024-05-15T13:15:14.248 INFO:tasks.workunit.client.0.smithi005.stdout: "last_active": "2024-05-15T13:14:49.873721+0000", 2024-05-15T13:15:14.248 INFO:tasks.workunit.client.0.smithi005.stdout: "last_peered": "2024-05-15T13:14:49.873721+0000", 2024-05-15T13:15:14.248 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean": "2024-05-15T13:14:16.946278+0000", 2024-05-15T13:15:14.248 INFO:tasks.workunit.client.0.smithi005.stdout: "last_became_active": "2024-05-15T13:14:19.225146+0000", 2024-05-15T13:15:14.248 INFO:tasks.workunit.client.0.smithi005.stdout: "last_became_peered": "2024-05-15T13:14:19.225146+0000", 2024-05-15T13:15:14.248 INFO:tasks.workunit.client.0.smithi005.stdout: "last_unstale": "2024-05-15T13:14:49.873721+0000", 2024-05-15T13:15:14.248 INFO:tasks.workunit.client.0.smithi005.stdout: "last_undegraded": "2024-05-15T13:14:18.809096+0000", 2024-05-15T13:15:14.248 INFO:tasks.workunit.client.0.smithi005.stdout: "last_fullsized": "2024-05-15T13:14:18.808870+0000", 2024-05-15T13:15:14.248 INFO:tasks.workunit.client.0.smithi005.stdout: "mapping_epoch": 55, 2024-05-15T13:15:14.248 INFO:tasks.workunit.client.0.smithi005.stdout: "log_start": "0'0", 2024-05-15T13:15:14.248 INFO:tasks.workunit.client.0.smithi005.stdout: "ondisk_log_start": "0'0", 2024-05-15T13:15:14.248 INFO:tasks.workunit.client.0.smithi005.stdout: "created": 43, 2024-05-15T13:15:14.248 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_clean": 44, 2024-05-15T13:15:14.248 INFO:tasks.workunit.client.0.smithi005.stdout: "parent": "0.0", 2024-05-15T13:15:14.248 INFO:tasks.workunit.client.0.smithi005.stdout: "parent_split_bits": 0, 2024-05-15T13:15:14.249 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub": "0'0", 2024-05-15T13:15:14.249 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.249 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub": "0'0", 2024-05-15T13:15:14.249 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.249 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.249 INFO:tasks.workunit.client.0.smithi005.stdout: "objects_scrubbed": 0, 2024-05-15T13:15:14.249 INFO:tasks.workunit.client.0.smithi005.stdout: "log_size": 101, 2024-05-15T13:15:14.249 INFO:tasks.workunit.client.0.smithi005.stdout: "log_dups_size": 0, 2024-05-15T13:15:14.249 INFO:tasks.workunit.client.0.smithi005.stdout: "ondisk_log_size": 101, 2024-05-15T13:15:14.249 INFO:tasks.workunit.client.0.smithi005.stdout: "stats_invalid": false, 2024-05-15T13:15:14.249 INFO:tasks.workunit.client.0.smithi005.stdout: "dirty_stats_invalid": false, 2024-05-15T13:15:14.249 INFO:tasks.workunit.client.0.smithi005.stdout: "omap_stats_invalid": false, 2024-05-15T13:15:14.249 INFO:tasks.workunit.client.0.smithi005.stdout: "hitset_stats_invalid": false, 2024-05-15T13:15:14.249 INFO:tasks.workunit.client.0.smithi005.stdout: "hitset_bytes_stats_invalid": false, 2024-05-15T13:15:14.249 INFO:tasks.workunit.client.0.smithi005.stdout: "pin_stats_invalid": false, 2024-05-15T13:15:14.249 INFO:tasks.workunit.client.0.smithi005.stdout: "manifest_stats_invalid": false, 2024-05-15T13:15:14.250 INFO:tasks.workunit.client.0.smithi005.stdout: "snaptrimq_len": 0, 2024-05-15T13:15:14.250 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_duration": 0, 2024-05-15T13:15:14.250 INFO:tasks.workunit.client.0.smithi005.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-05-16T19:15:16.210930+0000", 2024-05-15T13:15:14.250 INFO:tasks.workunit.client.0.smithi005.stdout: "scrub_duration": 0, 2024-05-15T13:15:14.250 INFO:tasks.workunit.client.0.smithi005.stdout: "objects_trimmed": 0, 2024-05-15T13:15:14.250 INFO:tasks.workunit.client.0.smithi005.stdout: "snaptrim_duration": 0, 2024-05-15T13:15:14.250 INFO:tasks.workunit.client.0.smithi005.stdout: "stat_sum": { 2024-05-15T13:15:14.250 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes": 413696, 2024-05-15T13:15:14.250 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects": 101, 2024-05-15T13:15:14.250 INFO:tasks.workunit.client.0.smithi005.stdout: "num_object_clones": 0, 2024-05-15T13:15:14.251 INFO:tasks.workunit.client.0.smithi005.stdout: "num_object_copies": 505, 2024-05-15T13:15:14.251 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_missing_on_primary": 0, 2024-05-15T13:15:14.251 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_missing": 1, 2024-05-15T13:15:14.251 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_degraded": 101, 2024-05-15T13:15:14.251 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_misplaced": 0, 2024-05-15T13:15:14.251 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_unfound": 0, 2024-05-15T13:15:14.251 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_dirty": 101, 2024-05-15T13:15:14.251 INFO:tasks.workunit.client.0.smithi005.stdout: "num_whiteouts": 0, 2024-05-15T13:15:14.251 INFO:tasks.workunit.client.0.smithi005.stdout: "num_read": 0, 2024-05-15T13:15:14.251 INFO:tasks.workunit.client.0.smithi005.stdout: "num_read_kb": 0, 2024-05-15T13:15:14.251 INFO:tasks.workunit.client.0.smithi005.stdout: "num_write": 101, 2024-05-15T13:15:14.251 INFO:tasks.workunit.client.0.smithi005.stdout: "num_write_kb": 404, 2024-05-15T13:15:14.251 INFO:tasks.workunit.client.0.smithi005.stdout: "num_scrub_errors": 0, 2024-05-15T13:15:14.251 INFO:tasks.workunit.client.0.smithi005.stdout: "num_shallow_scrub_errors": 0, 2024-05-15T13:15:14.251 INFO:tasks.workunit.client.0.smithi005.stdout: "num_deep_scrub_errors": 0, 2024-05-15T13:15:14.252 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_recovered": 0, 2024-05-15T13:15:14.252 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes_recovered": 0, 2024-05-15T13:15:14.252 INFO:tasks.workunit.client.0.smithi005.stdout: "num_keys_recovered": 0, 2024-05-15T13:15:14.252 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_omap": 0, 2024-05-15T13:15:14.252 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_hit_set_archive": 0, 2024-05-15T13:15:14.252 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes_hit_set_archive": 0, 2024-05-15T13:15:14.252 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush": 0, 2024-05-15T13:15:14.252 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_kb": 0, 2024-05-15T13:15:14.252 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict": 0, 2024-05-15T13:15:14.252 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_kb": 0, 2024-05-15T13:15:14.252 INFO:tasks.workunit.client.0.smithi005.stdout: "num_promote": 0, 2024-05-15T13:15:14.252 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_mode_high": 0, 2024-05-15T13:15:14.252 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_mode_low": 0, 2024-05-15T13:15:14.252 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_mode_some": 0, 2024-05-15T13:15:14.252 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_mode_full": 0, 2024-05-15T13:15:14.252 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_pinned": 0, 2024-05-15T13:15:14.253 INFO:tasks.workunit.client.0.smithi005.stdout: "num_legacy_snapsets": 0, 2024-05-15T13:15:14.253 INFO:tasks.workunit.client.0.smithi005.stdout: "num_large_omap_objects": 0, 2024-05-15T13:15:14.253 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_manifest": 0, 2024-05-15T13:15:14.253 INFO:tasks.workunit.client.0.smithi005.stdout: "num_omap_bytes": 0, 2024-05-15T13:15:14.253 INFO:tasks.workunit.client.0.smithi005.stdout: "num_omap_keys": 0, 2024-05-15T13:15:14.253 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_repaired": 0 2024-05-15T13:15:14.253 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.253 INFO:tasks.workunit.client.0.smithi005.stdout: "up": [ 2024-05-15T13:15:14.253 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T13:15:14.253 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T13:15:14.253 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T13:15:14.253 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T13:15:14.253 INFO:tasks.workunit.client.0.smithi005.stdout: 2 2024-05-15T13:15:14.253 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.253 INFO:tasks.workunit.client.0.smithi005.stdout: "acting": [ 2024-05-15T13:15:14.253 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T13:15:14.254 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T13:15:14.254 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T13:15:14.254 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T13:15:14.254 INFO:tasks.workunit.client.0.smithi005.stdout: 2 2024-05-15T13:15:14.254 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.254 INFO:tasks.workunit.client.0.smithi005.stdout: "avail_no_missing": [ 2024-05-15T13:15:14.254 INFO:tasks.workunit.client.0.smithi005.stdout: "3(0)", 2024-05-15T13:15:14.254 INFO:tasks.workunit.client.0.smithi005.stdout: "0(3)", 2024-05-15T13:15:14.254 INFO:tasks.workunit.client.0.smithi005.stdout: "1(1)", 2024-05-15T13:15:14.254 INFO:tasks.workunit.client.0.smithi005.stdout: "4(2)" 2024-05-15T13:15:14.254 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.254 INFO:tasks.workunit.client.0.smithi005.stdout: "object_location_counts": [ 2024-05-15T13:15:14.254 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T13:15:14.255 INFO:tasks.workunit.client.0.smithi005.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-05-15T13:15:14.255 INFO:tasks.workunit.client.0.smithi005.stdout: "objects": 101 2024-05-15T13:15:14.255 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T13:15:14.255 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.255 INFO:tasks.workunit.client.0.smithi005.stdout: "blocked_by": [], 2024-05-15T13:15:14.255 INFO:tasks.workunit.client.0.smithi005.stdout: "up_primary": 3, 2024-05-15T13:15:14.255 INFO:tasks.workunit.client.0.smithi005.stdout: "acting_primary": 3, 2024-05-15T13:15:14.255 INFO:tasks.workunit.client.0.smithi005.stdout: "purged_snaps": [] 2024-05-15T13:15:14.255 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.255 INFO:tasks.workunit.client.0.smithi005.stdout: "empty": 0, 2024-05-15T13:15:14.255 INFO:tasks.workunit.client.0.smithi005.stdout: "dne": 0, 2024-05-15T13:15:14.255 INFO:tasks.workunit.client.0.smithi005.stdout: "incomplete": 0, 2024-05-15T13:15:14.255 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_started": 57, 2024-05-15T13:15:14.255 INFO:tasks.workunit.client.0.smithi005.stdout: "hit_set_history": { 2024-05-15T13:15:14.256 INFO:tasks.workunit.client.0.smithi005.stdout: "current_last_update": "0'0", 2024-05-15T13:15:14.256 INFO:tasks.workunit.client.0.smithi005.stdout: "history": [] 2024-05-15T13:15:14.256 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T13:15:14.256 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.256 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T13:15:14.256 INFO:tasks.workunit.client.0.smithi005.stdout: "peer": "2(4)", 2024-05-15T13:15:14.256 INFO:tasks.workunit.client.0.smithi005.stdout: "pgid": "2.0s4", 2024-05-15T13:15:14.256 INFO:tasks.workunit.client.0.smithi005.stdout: "shared": "4", 2024-05-15T13:15:14.256 INFO:tasks.workunit.client.0.smithi005.stdout: "last_update": "51'101", 2024-05-15T13:15:14.256 INFO:tasks.workunit.client.0.smithi005.stdout: "last_complete": "45'1", 2024-05-15T13:15:14.256 INFO:tasks.workunit.client.0.smithi005.stdout: "log_tail": "0'0", 2024-05-15T13:15:14.256 INFO:tasks.workunit.client.0.smithi005.stdout: "last_user_version": 1, 2024-05-15T13:15:14.256 INFO:tasks.workunit.client.0.smithi005.stdout: "last_backfill": "MAX", 2024-05-15T13:15:14.256 INFO:tasks.workunit.client.0.smithi005.stdout: "purged_snaps": [], 2024-05-15T13:15:14.256 INFO:tasks.workunit.client.0.smithi005.stdout: "history": { 2024-05-15T13:15:14.256 INFO:tasks.workunit.client.0.smithi005.stdout: "epoch_created": 43, 2024-05-15T13:15:14.257 INFO:tasks.workunit.client.0.smithi005.stdout: "epoch_pool_created": 43, 2024-05-15T13:15:14.257 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_started": 57, 2024-05-15T13:15:14.257 INFO:tasks.workunit.client.0.smithi005.stdout: "last_interval_started": 55, 2024-05-15T13:15:14.257 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_clean": 44, 2024-05-15T13:15:14.257 INFO:tasks.workunit.client.0.smithi005.stdout: "last_interval_clean": 43, 2024-05-15T13:15:14.257 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_split": 0, 2024-05-15T13:15:14.257 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_marked_full": 0, 2024-05-15T13:15:14.257 INFO:tasks.workunit.client.0.smithi005.stdout: "same_up_since": 55, 2024-05-15T13:15:14.257 INFO:tasks.workunit.client.0.smithi005.stdout: "same_interval_since": 55, 2024-05-15T13:15:14.257 INFO:tasks.workunit.client.0.smithi005.stdout: "same_primary_since": 43, 2024-05-15T13:15:14.257 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub": "0'0", 2024-05-15T13:15:14.257 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.257 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub": "0'0", 2024-05-15T13:15:14.257 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.257 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.257 INFO:tasks.workunit.client.0.smithi005.stdout: "prior_readable_until_ub": 0 2024-05-15T13:15:14.257 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.258 INFO:tasks.workunit.client.0.smithi005.stdout: "stats": { 2024-05-15T13:15:14.258 INFO:tasks.workunit.client.0.smithi005.stdout: "version": "45'1", 2024-05-15T13:15:14.258 INFO:tasks.workunit.client.0.smithi005.stdout: "reported_seq": 17, 2024-05-15T13:15:14.258 INFO:tasks.workunit.client.0.smithi005.stdout: "reported_epoch": 45, 2024-05-15T13:15:14.258 INFO:tasks.workunit.client.0.smithi005.stdout: "state": "active+clean", 2024-05-15T13:15:14.258 INFO:tasks.workunit.client.0.smithi005.stdout: "last_fresh": "2024-05-15T13:14:16.946278+0000", 2024-05-15T13:15:14.258 INFO:tasks.workunit.client.0.smithi005.stdout: "last_change": "2024-05-15T13:14:10.114170+0000", 2024-05-15T13:15:14.258 INFO:tasks.workunit.client.0.smithi005.stdout: "last_active": "2024-05-15T13:14:16.946278+0000", 2024-05-15T13:15:14.258 INFO:tasks.workunit.client.0.smithi005.stdout: "last_peered": "2024-05-15T13:14:16.946278+0000", 2024-05-15T13:15:14.258 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean": "2024-05-15T13:14:16.946278+0000", 2024-05-15T13:15:14.258 INFO:tasks.workunit.client.0.smithi005.stdout: "last_became_active": "2024-05-15T13:14:10.112981+0000", 2024-05-15T13:15:14.258 INFO:tasks.workunit.client.0.smithi005.stdout: "last_became_peered": "2024-05-15T13:14:10.112981+0000", 2024-05-15T13:15:14.259 INFO:tasks.workunit.client.0.smithi005.stdout: "last_unstale": "2024-05-15T13:14:16.946278+0000", 2024-05-15T13:15:14.259 INFO:tasks.workunit.client.0.smithi005.stdout: "last_undegraded": "2024-05-15T13:14:16.946278+0000", 2024-05-15T13:15:14.259 INFO:tasks.workunit.client.0.smithi005.stdout: "last_fullsized": "2024-05-15T13:14:16.946278+0000", 2024-05-15T13:15:14.259 INFO:tasks.workunit.client.0.smithi005.stdout: "mapping_epoch": 55, 2024-05-15T13:15:14.259 INFO:tasks.workunit.client.0.smithi005.stdout: "log_start": "0'0", 2024-05-15T13:15:14.259 INFO:tasks.workunit.client.0.smithi005.stdout: "ondisk_log_start": "0'0", 2024-05-15T13:15:14.259 INFO:tasks.workunit.client.0.smithi005.stdout: "created": 43, 2024-05-15T13:15:14.259 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_clean": 44, 2024-05-15T13:15:14.259 INFO:tasks.workunit.client.0.smithi005.stdout: "parent": "0.0", 2024-05-15T13:15:14.259 INFO:tasks.workunit.client.0.smithi005.stdout: "parent_split_bits": 0, 2024-05-15T13:15:14.259 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub": "0'0", 2024-05-15T13:15:14.259 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.259 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub": "0'0", 2024-05-15T13:15:14.259 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.259 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.259 INFO:tasks.workunit.client.0.smithi005.stdout: "objects_scrubbed": 0, 2024-05-15T13:15:14.260 INFO:tasks.workunit.client.0.smithi005.stdout: "log_size": 1, 2024-05-15T13:15:14.260 INFO:tasks.workunit.client.0.smithi005.stdout: "log_dups_size": 0, 2024-05-15T13:15:14.260 INFO:tasks.workunit.client.0.smithi005.stdout: "ondisk_log_size": 1, 2024-05-15T13:15:14.260 INFO:tasks.workunit.client.0.smithi005.stdout: "stats_invalid": false, 2024-05-15T13:15:14.260 INFO:tasks.workunit.client.0.smithi005.stdout: "dirty_stats_invalid": false, 2024-05-15T13:15:14.260 INFO:tasks.workunit.client.0.smithi005.stdout: "omap_stats_invalid": false, 2024-05-15T13:15:14.260 INFO:tasks.workunit.client.0.smithi005.stdout: "hitset_stats_invalid": false, 2024-05-15T13:15:14.260 INFO:tasks.workunit.client.0.smithi005.stdout: "hitset_bytes_stats_invalid": false, 2024-05-15T13:15:14.260 INFO:tasks.workunit.client.0.smithi005.stdout: "pin_stats_invalid": false, 2024-05-15T13:15:14.260 INFO:tasks.workunit.client.0.smithi005.stdout: "manifest_stats_invalid": false, 2024-05-15T13:15:14.260 INFO:tasks.workunit.client.0.smithi005.stdout: "snaptrimq_len": 0, 2024-05-15T13:15:14.260 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_duration": 0, 2024-05-15T13:15:14.260 INFO:tasks.workunit.client.0.smithi005.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-05-16T18:30:38.016564+0000", 2024-05-15T13:15:14.260 INFO:tasks.workunit.client.0.smithi005.stdout: "scrub_duration": 0, 2024-05-15T13:15:14.260 INFO:tasks.workunit.client.0.smithi005.stdout: "objects_trimmed": 0, 2024-05-15T13:15:14.261 INFO:tasks.workunit.client.0.smithi005.stdout: "snaptrim_duration": 0, 2024-05-15T13:15:14.261 INFO:tasks.workunit.client.0.smithi005.stdout: "stat_sum": { 2024-05-15T13:15:14.261 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes": 4096, 2024-05-15T13:15:14.261 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects": 1, 2024-05-15T13:15:14.261 INFO:tasks.workunit.client.0.smithi005.stdout: "num_object_clones": 0, 2024-05-15T13:15:14.261 INFO:tasks.workunit.client.0.smithi005.stdout: "num_object_copies": 5, 2024-05-15T13:15:14.261 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_missing_on_primary": 0, 2024-05-15T13:15:14.261 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_missing": 1, 2024-05-15T13:15:14.261 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_degraded": 0, 2024-05-15T13:15:14.261 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_misplaced": 0, 2024-05-15T13:15:14.261 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_unfound": 0, 2024-05-15T13:15:14.261 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_dirty": 1, 2024-05-15T13:15:14.262 INFO:tasks.workunit.client.0.smithi005.stdout: "num_whiteouts": 0, 2024-05-15T13:15:14.262 INFO:tasks.workunit.client.0.smithi005.stdout: "num_read": 0, 2024-05-15T13:15:14.262 INFO:tasks.workunit.client.0.smithi005.stdout: "num_read_kb": 0, 2024-05-15T13:15:14.262 INFO:tasks.workunit.client.0.smithi005.stdout: "num_write": 1, 2024-05-15T13:15:14.262 INFO:tasks.workunit.client.0.smithi005.stdout: "num_write_kb": 4, 2024-05-15T13:15:14.262 INFO:tasks.workunit.client.0.smithi005.stdout: "num_scrub_errors": 0, 2024-05-15T13:15:14.262 INFO:tasks.workunit.client.0.smithi005.stdout: "num_shallow_scrub_errors": 0, 2024-05-15T13:15:14.262 INFO:tasks.workunit.client.0.smithi005.stdout: "num_deep_scrub_errors": 0, 2024-05-15T13:15:14.262 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_recovered": 0, 2024-05-15T13:15:14.262 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes_recovered": 0, 2024-05-15T13:15:14.262 INFO:tasks.workunit.client.0.smithi005.stdout: "num_keys_recovered": 0, 2024-05-15T13:15:14.262 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_omap": 0, 2024-05-15T13:15:14.263 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_hit_set_archive": 0, 2024-05-15T13:15:14.263 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes_hit_set_archive": 0, 2024-05-15T13:15:14.263 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush": 0, 2024-05-15T13:15:14.263 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_kb": 0, 2024-05-15T13:15:14.263 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict": 0, 2024-05-15T13:15:14.263 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_kb": 0, 2024-05-15T13:15:14.263 INFO:tasks.workunit.client.0.smithi005.stdout: "num_promote": 0, 2024-05-15T13:15:14.263 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_mode_high": 0, 2024-05-15T13:15:14.263 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_mode_low": 0, 2024-05-15T13:15:14.263 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_mode_some": 0, 2024-05-15T13:15:14.263 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_mode_full": 0, 2024-05-15T13:15:14.263 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_pinned": 0, 2024-05-15T13:15:14.263 INFO:tasks.workunit.client.0.smithi005.stdout: "num_legacy_snapsets": 0, 2024-05-15T13:15:14.264 INFO:tasks.workunit.client.0.smithi005.stdout: "num_large_omap_objects": 0, 2024-05-15T13:15:14.264 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_manifest": 0, 2024-05-15T13:15:14.264 INFO:tasks.workunit.client.0.smithi005.stdout: "num_omap_bytes": 0, 2024-05-15T13:15:14.264 INFO:tasks.workunit.client.0.smithi005.stdout: "num_omap_keys": 0, 2024-05-15T13:15:14.264 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_repaired": 0 2024-05-15T13:15:14.264 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.264 INFO:tasks.workunit.client.0.smithi005.stdout: "up": [ 2024-05-15T13:15:14.264 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T13:15:14.264 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T13:15:14.264 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T13:15:14.265 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T13:15:14.265 INFO:tasks.workunit.client.0.smithi005.stdout: 2 2024-05-15T13:15:14.265 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.265 INFO:tasks.workunit.client.0.smithi005.stdout: "acting": [ 2024-05-15T13:15:14.265 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T13:15:14.265 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T13:15:14.265 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T13:15:14.265 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T13:15:14.265 INFO:tasks.workunit.client.0.smithi005.stdout: 2 2024-05-15T13:15:14.265 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.265 INFO:tasks.workunit.client.0.smithi005.stdout: "avail_no_missing": [], 2024-05-15T13:15:14.265 INFO:tasks.workunit.client.0.smithi005.stdout: "object_location_counts": [], 2024-05-15T13:15:14.266 INFO:tasks.workunit.client.0.smithi005.stdout: "blocked_by": [], 2024-05-15T13:15:14.266 INFO:tasks.workunit.client.0.smithi005.stdout: "up_primary": 3, 2024-05-15T13:15:14.266 INFO:tasks.workunit.client.0.smithi005.stdout: "acting_primary": 3, 2024-05-15T13:15:14.266 INFO:tasks.workunit.client.0.smithi005.stdout: "purged_snaps": [] 2024-05-15T13:15:14.266 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.266 INFO:tasks.workunit.client.0.smithi005.stdout: "empty": 0, 2024-05-15T13:15:14.266 INFO:tasks.workunit.client.0.smithi005.stdout: "dne": 0, 2024-05-15T13:15:14.266 INFO:tasks.workunit.client.0.smithi005.stdout: "incomplete": 0, 2024-05-15T13:15:14.266 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_started": 57, 2024-05-15T13:15:14.266 INFO:tasks.workunit.client.0.smithi005.stdout: "hit_set_history": { 2024-05-15T13:15:14.266 INFO:tasks.workunit.client.0.smithi005.stdout: "current_last_update": "0'0", 2024-05-15T13:15:14.266 INFO:tasks.workunit.client.0.smithi005.stdout: "history": [] 2024-05-15T13:15:14.266 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T13:15:14.267 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.267 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T13:15:14.267 INFO:tasks.workunit.client.0.smithi005.stdout: "peer": "4(2)", 2024-05-15T13:15:14.267 INFO:tasks.workunit.client.0.smithi005.stdout: "pgid": "2.0s2", 2024-05-15T13:15:14.267 INFO:tasks.workunit.client.0.smithi005.stdout: "shared": "2", 2024-05-15T13:15:14.267 INFO:tasks.workunit.client.0.smithi005.stdout: "last_update": "51'101", 2024-05-15T13:15:14.267 INFO:tasks.workunit.client.0.smithi005.stdout: "last_complete": "51'101", 2024-05-15T13:15:14.267 INFO:tasks.workunit.client.0.smithi005.stdout: "log_tail": "0'0", 2024-05-15T13:15:14.267 INFO:tasks.workunit.client.0.smithi005.stdout: "last_user_version": 101, 2024-05-15T13:15:14.267 INFO:tasks.workunit.client.0.smithi005.stdout: "last_backfill": "MAX", 2024-05-15T13:15:14.267 INFO:tasks.workunit.client.0.smithi005.stdout: "purged_snaps": [], 2024-05-15T13:15:14.268 INFO:tasks.workunit.client.0.smithi005.stdout: "history": { 2024-05-15T13:15:14.268 INFO:tasks.workunit.client.0.smithi005.stdout: "epoch_created": 43, 2024-05-15T13:15:14.268 INFO:tasks.workunit.client.0.smithi005.stdout: "epoch_pool_created": 43, 2024-05-15T13:15:14.268 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_started": 57, 2024-05-15T13:15:14.268 INFO:tasks.workunit.client.0.smithi005.stdout: "last_interval_started": 55, 2024-05-15T13:15:14.268 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_clean": 44, 2024-05-15T13:15:14.268 INFO:tasks.workunit.client.0.smithi005.stdout: "last_interval_clean": 43, 2024-05-15T13:15:14.268 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_split": 0, 2024-05-15T13:15:14.268 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_marked_full": 0, 2024-05-15T13:15:14.268 INFO:tasks.workunit.client.0.smithi005.stdout: "same_up_since": 55, 2024-05-15T13:15:14.268 INFO:tasks.workunit.client.0.smithi005.stdout: "same_interval_since": 55, 2024-05-15T13:15:14.268 INFO:tasks.workunit.client.0.smithi005.stdout: "same_primary_since": 43, 2024-05-15T13:15:14.269 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub": "0'0", 2024-05-15T13:15:14.269 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.269 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub": "0'0", 2024-05-15T13:15:14.269 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.269 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.269 INFO:tasks.workunit.client.0.smithi005.stdout: "prior_readable_until_ub": 0 2024-05-15T13:15:14.269 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.269 INFO:tasks.workunit.client.0.smithi005.stdout: "stats": { 2024-05-15T13:15:14.269 INFO:tasks.workunit.client.0.smithi005.stdout: "version": "51'101", 2024-05-15T13:15:14.269 INFO:tasks.workunit.client.0.smithi005.stdout: "reported_seq": 132, 2024-05-15T13:15:14.269 INFO:tasks.workunit.client.0.smithi005.stdout: "reported_epoch": 51, 2024-05-15T13:15:14.269 INFO:tasks.workunit.client.0.smithi005.stdout: "state": "active+undersized+degraded", 2024-05-15T13:15:14.269 INFO:tasks.workunit.client.0.smithi005.stdout: "last_fresh": "2024-05-15T13:14:49.873721+0000", 2024-05-15T13:15:14.269 INFO:tasks.workunit.client.0.smithi005.stdout: "last_change": "2024-05-15T13:14:19.225146+0000", 2024-05-15T13:15:14.269 INFO:tasks.workunit.client.0.smithi005.stdout: "last_active": "2024-05-15T13:14:49.873721+0000", 2024-05-15T13:15:14.270 INFO:tasks.workunit.client.0.smithi005.stdout: "last_peered": "2024-05-15T13:14:49.873721+0000", 2024-05-15T13:15:14.270 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean": "2024-05-15T13:14:16.946278+0000", 2024-05-15T13:15:14.270 INFO:tasks.workunit.client.0.smithi005.stdout: "last_became_active": "2024-05-15T13:14:19.225146+0000", 2024-05-15T13:15:14.270 INFO:tasks.workunit.client.0.smithi005.stdout: "last_became_peered": "2024-05-15T13:14:19.225146+0000", 2024-05-15T13:15:14.270 INFO:tasks.workunit.client.0.smithi005.stdout: "last_unstale": "2024-05-15T13:14:49.873721+0000", 2024-05-15T13:15:14.270 INFO:tasks.workunit.client.0.smithi005.stdout: "last_undegraded": "2024-05-15T13:14:18.809096+0000", 2024-05-15T13:15:14.270 INFO:tasks.workunit.client.0.smithi005.stdout: "last_fullsized": "2024-05-15T13:14:18.808870+0000", 2024-05-15T13:15:14.270 INFO:tasks.workunit.client.0.smithi005.stdout: "mapping_epoch": 55, 2024-05-15T13:15:14.270 INFO:tasks.workunit.client.0.smithi005.stdout: "log_start": "0'0", 2024-05-15T13:15:14.270 INFO:tasks.workunit.client.0.smithi005.stdout: "ondisk_log_start": "0'0", 2024-05-15T13:15:14.270 INFO:tasks.workunit.client.0.smithi005.stdout: "created": 43, 2024-05-15T13:15:14.270 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_clean": 44, 2024-05-15T13:15:14.270 INFO:tasks.workunit.client.0.smithi005.stdout: "parent": "0.0", 2024-05-15T13:15:14.271 INFO:tasks.workunit.client.0.smithi005.stdout: "parent_split_bits": 0, 2024-05-15T13:15:14.271 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub": "0'0", 2024-05-15T13:15:14.271 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.271 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub": "0'0", 2024-05-15T13:15:14.271 INFO:tasks.workunit.client.0.smithi005.stdout: "last_deep_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.271 INFO:tasks.workunit.client.0.smithi005.stdout: "last_clean_scrub_stamp": "2024-05-15T13:14:08.249128+0000", 2024-05-15T13:15:14.271 INFO:tasks.workunit.client.0.smithi005.stdout: "objects_scrubbed": 0, 2024-05-15T13:15:14.271 INFO:tasks.workunit.client.0.smithi005.stdout: "log_size": 101, 2024-05-15T13:15:14.271 INFO:tasks.workunit.client.0.smithi005.stdout: "log_dups_size": 0, 2024-05-15T13:15:14.271 INFO:tasks.workunit.client.0.smithi005.stdout: "ondisk_log_size": 101, 2024-05-15T13:15:14.272 INFO:tasks.workunit.client.0.smithi005.stdout: "stats_invalid": false, 2024-05-15T13:15:14.272 INFO:tasks.workunit.client.0.smithi005.stdout: "dirty_stats_invalid": false, 2024-05-15T13:15:14.272 INFO:tasks.workunit.client.0.smithi005.stdout: "omap_stats_invalid": false, 2024-05-15T13:15:14.272 INFO:tasks.workunit.client.0.smithi005.stdout: "hitset_stats_invalid": false, 2024-05-15T13:15:14.272 INFO:tasks.workunit.client.0.smithi005.stdout: "hitset_bytes_stats_invalid": false, 2024-05-15T13:15:14.272 INFO:tasks.workunit.client.0.smithi005.stdout: "pin_stats_invalid": false, 2024-05-15T13:15:14.272 INFO:tasks.workunit.client.0.smithi005.stdout: "manifest_stats_invalid": false, 2024-05-15T13:15:14.272 INFO:tasks.workunit.client.0.smithi005.stdout: "snaptrimq_len": 0, 2024-05-15T13:15:14.272 INFO:tasks.workunit.client.0.smithi005.stdout: "last_scrub_duration": 0, 2024-05-15T13:15:14.272 INFO:tasks.workunit.client.0.smithi005.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-05-16T19:15:16.210930+0000", 2024-05-15T13:15:14.272 INFO:tasks.workunit.client.0.smithi005.stdout: "scrub_duration": 0, 2024-05-15T13:15:14.272 INFO:tasks.workunit.client.0.smithi005.stdout: "objects_trimmed": 0, 2024-05-15T13:15:14.273 INFO:tasks.workunit.client.0.smithi005.stdout: "snaptrim_duration": 0, 2024-05-15T13:15:14.273 INFO:tasks.workunit.client.0.smithi005.stdout: "stat_sum": { 2024-05-15T13:15:14.273 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes": 413696, 2024-05-15T13:15:14.273 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects": 101, 2024-05-15T13:15:14.273 INFO:tasks.workunit.client.0.smithi005.stdout: "num_object_clones": 0, 2024-05-15T13:15:14.273 INFO:tasks.workunit.client.0.smithi005.stdout: "num_object_copies": 505, 2024-05-15T13:15:14.273 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_missing_on_primary": 0, 2024-05-15T13:15:14.273 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_missing": 0, 2024-05-15T13:15:14.273 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_degraded": 101, 2024-05-15T13:15:14.273 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_misplaced": 0, 2024-05-15T13:15:14.273 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_unfound": 0, 2024-05-15T13:15:14.273 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_dirty": 101, 2024-05-15T13:15:14.273 INFO:tasks.workunit.client.0.smithi005.stdout: "num_whiteouts": 0, 2024-05-15T13:15:14.273 INFO:tasks.workunit.client.0.smithi005.stdout: "num_read": 0, 2024-05-15T13:15:14.274 INFO:tasks.workunit.client.0.smithi005.stdout: "num_read_kb": 0, 2024-05-15T13:15:14.274 INFO:tasks.workunit.client.0.smithi005.stdout: "num_write": 101, 2024-05-15T13:15:14.274 INFO:tasks.workunit.client.0.smithi005.stdout: "num_write_kb": 404, 2024-05-15T13:15:14.274 INFO:tasks.workunit.client.0.smithi005.stdout: "num_scrub_errors": 0, 2024-05-15T13:15:14.274 INFO:tasks.workunit.client.0.smithi005.stdout: "num_shallow_scrub_errors": 0, 2024-05-15T13:15:14.274 INFO:tasks.workunit.client.0.smithi005.stdout: "num_deep_scrub_errors": 0, 2024-05-15T13:15:14.274 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_recovered": 0, 2024-05-15T13:15:14.274 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes_recovered": 0, 2024-05-15T13:15:14.274 INFO:tasks.workunit.client.0.smithi005.stdout: "num_keys_recovered": 0, 2024-05-15T13:15:14.274 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_omap": 0, 2024-05-15T13:15:14.274 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_hit_set_archive": 0, 2024-05-15T13:15:14.274 INFO:tasks.workunit.client.0.smithi005.stdout: "num_bytes_hit_set_archive": 0, 2024-05-15T13:15:14.274 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush": 0, 2024-05-15T13:15:14.274 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_kb": 0, 2024-05-15T13:15:14.275 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict": 0, 2024-05-15T13:15:14.275 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_kb": 0, 2024-05-15T13:15:14.275 INFO:tasks.workunit.client.0.smithi005.stdout: "num_promote": 0, 2024-05-15T13:15:14.275 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_mode_high": 0, 2024-05-15T13:15:14.275 INFO:tasks.workunit.client.0.smithi005.stdout: "num_flush_mode_low": 0, 2024-05-15T13:15:14.275 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_mode_some": 0, 2024-05-15T13:15:14.275 INFO:tasks.workunit.client.0.smithi005.stdout: "num_evict_mode_full": 0, 2024-05-15T13:15:14.275 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_pinned": 0, 2024-05-15T13:15:14.275 INFO:tasks.workunit.client.0.smithi005.stdout: "num_legacy_snapsets": 0, 2024-05-15T13:15:14.275 INFO:tasks.workunit.client.0.smithi005.stdout: "num_large_omap_objects": 0, 2024-05-15T13:15:14.275 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_manifest": 0, 2024-05-15T13:15:14.276 INFO:tasks.workunit.client.0.smithi005.stdout: "num_omap_bytes": 0, 2024-05-15T13:15:14.276 INFO:tasks.workunit.client.0.smithi005.stdout: "num_omap_keys": 0, 2024-05-15T13:15:14.276 INFO:tasks.workunit.client.0.smithi005.stdout: "num_objects_repaired": 0 2024-05-15T13:15:14.276 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.276 INFO:tasks.workunit.client.0.smithi005.stdout: "up": [ 2024-05-15T13:15:14.276 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T13:15:14.276 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T13:15:14.276 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T13:15:14.276 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T13:15:14.276 INFO:tasks.workunit.client.0.smithi005.stdout: 2 2024-05-15T13:15:14.276 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.276 INFO:tasks.workunit.client.0.smithi005.stdout: "acting": [ 2024-05-15T13:15:14.276 INFO:tasks.workunit.client.0.smithi005.stdout: 3, 2024-05-15T13:15:14.276 INFO:tasks.workunit.client.0.smithi005.stdout: 1, 2024-05-15T13:15:14.276 INFO:tasks.workunit.client.0.smithi005.stdout: 4, 2024-05-15T13:15:14.276 INFO:tasks.workunit.client.0.smithi005.stdout: 0, 2024-05-15T13:15:14.276 INFO:tasks.workunit.client.0.smithi005.stdout: 2 2024-05-15T13:15:14.277 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.277 INFO:tasks.workunit.client.0.smithi005.stdout: "avail_no_missing": [ 2024-05-15T13:15:14.277 INFO:tasks.workunit.client.0.smithi005.stdout: "3(0)", 2024-05-15T13:15:14.277 INFO:tasks.workunit.client.0.smithi005.stdout: "0(3)", 2024-05-15T13:15:14.277 INFO:tasks.workunit.client.0.smithi005.stdout: "1(1)", 2024-05-15T13:15:14.277 INFO:tasks.workunit.client.0.smithi005.stdout: "4(2)" 2024-05-15T13:15:14.277 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.277 INFO:tasks.workunit.client.0.smithi005.stdout: "object_location_counts": [ 2024-05-15T13:15:14.277 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T13:15:14.277 INFO:tasks.workunit.client.0.smithi005.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-05-15T13:15:14.277 INFO:tasks.workunit.client.0.smithi005.stdout: "objects": 101 2024-05-15T13:15:14.277 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T13:15:14.277 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.277 INFO:tasks.workunit.client.0.smithi005.stdout: "blocked_by": [], 2024-05-15T13:15:14.277 INFO:tasks.workunit.client.0.smithi005.stdout: "up_primary": 3, 2024-05-15T13:15:14.277 INFO:tasks.workunit.client.0.smithi005.stdout: "acting_primary": 3, 2024-05-15T13:15:14.277 INFO:tasks.workunit.client.0.smithi005.stdout: "purged_snaps": [] 2024-05-15T13:15:14.278 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.278 INFO:tasks.workunit.client.0.smithi005.stdout: "empty": 0, 2024-05-15T13:15:14.278 INFO:tasks.workunit.client.0.smithi005.stdout: "dne": 0, 2024-05-15T13:15:14.278 INFO:tasks.workunit.client.0.smithi005.stdout: "incomplete": 0, 2024-05-15T13:15:14.278 INFO:tasks.workunit.client.0.smithi005.stdout: "last_epoch_started": 57, 2024-05-15T13:15:14.278 INFO:tasks.workunit.client.0.smithi005.stdout: "hit_set_history": { 2024-05-15T13:15:14.278 INFO:tasks.workunit.client.0.smithi005.stdout: "current_last_update": "0'0", 2024-05-15T13:15:14.278 INFO:tasks.workunit.client.0.smithi005.stdout: "history": [] 2024-05-15T13:15:14.278 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T13:15:14.278 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T13:15:14.278 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.278 INFO:tasks.workunit.client.0.smithi005.stdout: "recovery_state": [ 2024-05-15T13:15:14.278 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T13:15:14.279 INFO:tasks.workunit.client.0.smithi005.stdout: "name": "Started/Primary/Active", 2024-05-15T13:15:14.279 INFO:tasks.workunit.client.0.smithi005.stdout: "enter_time": "2024-05-15T13:14:58.641081+0000", 2024-05-15T13:15:14.279 INFO:tasks.workunit.client.0.smithi005.stdout: "might_have_unfound": [ 2024-05-15T13:15:14.279 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T13:15:14.279 INFO:tasks.workunit.client.0.smithi005.stdout: "osd": "0(3)", 2024-05-15T13:15:14.279 INFO:tasks.workunit.client.0.smithi005.stdout: "status": "already probed" 2024-05-15T13:15:14.279 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.279 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T13:15:14.279 INFO:tasks.workunit.client.0.smithi005.stdout: "osd": "1(1)", 2024-05-15T13:15:14.279 INFO:tasks.workunit.client.0.smithi005.stdout: "status": "already probed" 2024-05-15T13:15:14.279 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.279 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T13:15:14.279 INFO:tasks.workunit.client.0.smithi005.stdout: "osd": "2(4)", 2024-05-15T13:15:14.279 INFO:tasks.workunit.client.0.smithi005.stdout: "status": "already probed" 2024-05-15T13:15:14.279 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.279 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T13:15:14.279 INFO:tasks.workunit.client.0.smithi005.stdout: "osd": "4(2)", 2024-05-15T13:15:14.280 INFO:tasks.workunit.client.0.smithi005.stdout: "status": "already probed" 2024-05-15T13:15:14.280 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T13:15:14.280 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.280 INFO:tasks.workunit.client.0.smithi005.stdout: "recovery_progress": { 2024-05-15T13:15:14.280 INFO:tasks.workunit.client.0.smithi005.stdout: "backfill_targets": [], 2024-05-15T13:15:14.280 INFO:tasks.workunit.client.0.smithi005.stdout: "waiting_on_backfill": [], 2024-05-15T13:15:14.280 INFO:tasks.workunit.client.0.smithi005.stdout: "last_backfill_started": "MIN", 2024-05-15T13:15:14.280 INFO:tasks.workunit.client.0.smithi005.stdout: "backfill_info": { 2024-05-15T13:15:14.280 INFO:tasks.workunit.client.0.smithi005.stdout: "begin": "MIN", 2024-05-15T13:15:14.280 INFO:tasks.workunit.client.0.smithi005.stdout: "end": "MIN", 2024-05-15T13:15:14.280 INFO:tasks.workunit.client.0.smithi005.stdout: "objects": [] 2024-05-15T13:15:14.280 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.280 INFO:tasks.workunit.client.0.smithi005.stdout: "peer_backfill_info": [], 2024-05-15T13:15:14.280 INFO:tasks.workunit.client.0.smithi005.stdout: "backfills_in_flight": [], 2024-05-15T13:15:14.280 INFO:tasks.workunit.client.0.smithi005.stdout: "recovering": [], 2024-05-15T13:15:14.280 INFO:tasks.workunit.client.0.smithi005.stdout: "pg_backend": { 2024-05-15T13:15:14.280 INFO:tasks.workunit.client.0.smithi005.stdout: "recovery_ops": [], 2024-05-15T13:15:14.280 INFO:tasks.workunit.client.0.smithi005.stdout: "read_ops": [] 2024-05-15T13:15:14.281 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T13:15:14.281 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T13:15:14.281 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.281 INFO:tasks.workunit.client.0.smithi005.stdout: { 2024-05-15T13:15:14.281 INFO:tasks.workunit.client.0.smithi005.stdout: "name": "Started", 2024-05-15T13:15:14.281 INFO:tasks.workunit.client.0.smithi005.stdout: "enter_time": "2024-05-15T13:14:57.758611+0000" 2024-05-15T13:15:14.281 INFO:tasks.workunit.client.0.smithi005.stdout: } 2024-05-15T13:15:14.281 INFO:tasks.workunit.client.0.smithi005.stdout: ], 2024-05-15T13:15:14.281 INFO:tasks.workunit.client.0.smithi005.stdout: "scrubber": { 2024-05-15T13:15:14.281 INFO:tasks.workunit.client.0.smithi005.stdout: "active": false, 2024-05-15T13:15:14.281 INFO:tasks.workunit.client.0.smithi005.stdout: "must_scrub": false, 2024-05-15T13:15:14.281 INFO:tasks.workunit.client.0.smithi005.stdout: "must_deep_scrub": false, 2024-05-15T13:15:14.281 INFO:tasks.workunit.client.0.smithi005.stdout: "must_repair": false, 2024-05-15T13:15:14.281 INFO:tasks.workunit.client.0.smithi005.stdout: "need_auto": false, 2024-05-15T13:15:14.281 INFO:tasks.workunit.client.0.smithi005.stdout: "scrub_reg_stamp": "2024-05-16T19:15:16.210930+0000", 2024-05-15T13:15:14.281 INFO:tasks.workunit.client.0.smithi005.stdout: "schedule": "no scrub is scheduled" 2024-05-15T13:15:14.281 INFO:tasks.workunit.client.0.smithi005.stdout: }, 2024-05-15T13:15:14.282 INFO:tasks.workunit.client.0.smithi005.stdout: "agent_state": {} 2024-05-15T13:15:14.282 INFO:tasks.workunit.client.0.smithi005.stdout:} 2024-05-15T13:15:14.282 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:664: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-05-15T13:15:14.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:664: TEST_ec_recovery_unfound: grep -q obj75 2024-05-15T13:15:14.389 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: jq .available_might_have_unfound 2024-05-15T13:15:14.390 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-05-15T13:15:14.552 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: check=true 2024-05-15T13:15:14.552 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:667: TEST_ec_recovery_unfound: test true == true 2024-05-15T13:15:14.553 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-05-15T13:15:14.553 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: jq '.might_have_unfound | length' 2024-05-15T13:15:14.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: check=0 2024-05-15T13:15:14.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:670: TEST_ec_recovery_unfound: test 0 == 0 2024-05-15T13:15:14.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:673: TEST_ec_recovery_unfound: timeout 5 rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-05-15T13:15:19.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:674: TEST_ec_recovery_unfound: test 124 = 124 2024-05-15T13:15:19.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:676: TEST_ec_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-05-15T13:15:20.034 INFO:tasks.workunit.client.0.smithi005.stderr:pg has no unfound objects 2024-05-15T13:15:20.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:678: TEST_ec_recovery_unfound: wait_for_clean 2024-05-15T13:15:20.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:15:20.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:15:20.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:15:20.057 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:15:20.058 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:15:20.058 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:15:20.058 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:15:20.058 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:15:20.058 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:15:20.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:15:20.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:15:20.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:15:20.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:15:20.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:15:20.185 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:15:20.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:15:20.711 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:15:20.712 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:15:20.712 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T13:15:20.712 INFO:tasks.workunit.client.0.smithi005.stderr:4' 2024-05-15T13:15:20.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:15:20.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:15:20.712 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:15:20.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803832 2024-05-15T13:15:20.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803832 2024-05-15T13:15:20.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803832' 2024-05-15T13:15:20.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:15:20.876 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:15:21.030 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574896 2024-05-15T13:15:21.030 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574896 2024-05-15T13:15:21.030 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803832 1-55834574896' 2024-05-15T13:15:21.030 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:15:21.030 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:15:21.190 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266695 2024-05-15T13:15:21.190 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266695 2024-05-15T13:15:21.191 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803832 1-55834574896 2-227633266695' 2024-05-15T13:15:21.191 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:15:21.191 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:15:21.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117022 2024-05-15T13:15:21.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117022 2024-05-15T13:15:21.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803832 1-55834574896 2-227633266695 3-115964117022' 2024-05-15T13:15:21.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:15:21.355 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T13:15:21.511 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789981 2024-05-15T13:15:21.511 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789981 2024-05-15T13:15:21.511 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803832 1-55834574896 2-227633266695 3-115964117022 4-158913789981' 2024-05-15T13:15:21.511 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:15:21.511 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803832 2024-05-15T13:15:21.512 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:15:21.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:15:21.513 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803832 2024-05-15T13:15:21.513 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:15:21.514 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803832 2024-05-15T13:15:21.515 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 25769803832 2024-05-15T13:15:21.515 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803832' 2024-05-15T13:15:21.515 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:15:21.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803832 -lt 25769803832 2024-05-15T13:15:21.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:15:21.977 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574896 2024-05-15T13:15:21.977 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:15:21.978 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:15:21.979 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574896 2024-05-15T13:15:21.979 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:15:21.980 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574896 2024-05-15T13:15:21.981 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 55834574896 2024-05-15T13:15:21.981 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574896' 2024-05-15T13:15:21.981 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:15:22.442 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574896 -lt 55834574896 2024-05-15T13:15:22.442 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:15:22.443 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:15:22.443 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-227633266695 2024-05-15T13:15:22.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:15:22.446 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-227633266695 2024-05-15T13:15:22.446 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:15:22.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266695 2024-05-15T13:15:22.447 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 227633266695 2024-05-15T13:15:22.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 227633266695' 2024-05-15T13:15:22.447 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:15:22.948 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266696 -lt 227633266695 2024-05-15T13:15:22.948 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:15:22.949 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117022 2024-05-15T13:15:22.949 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:15:22.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:15:22.951 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117022 2024-05-15T13:15:22.951 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:15:22.952 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117022 2024-05-15T13:15:22.953 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 115964117022 2024-05-15T13:15:22.953 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117022' 2024-05-15T13:15:22.953 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:15:23.352 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117022 -lt 115964117022 2024-05-15T13:15:23.353 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:15:23.353 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:15:23.353 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-158913789981 2024-05-15T13:15:23.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T13:15:23.355 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-158913789981 2024-05-15T13:15:23.355 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:15:23.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789981 2024-05-15T13:15:23.356 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.4 seq 158913789981 2024-05-15T13:15:23.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 158913789981' 2024-05-15T13:15:23.357 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T13:15:23.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789981 -lt 158913789981 2024-05-15T13:15:23.767 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:15:23.767 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:15:23.768 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:15:24.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:15:24.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:15:24.435 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:15:24.436 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:15:24.436 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:15:24.436 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:15:24.436 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:15:24.436 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:15:24.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:15:24.877 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:15:24.877 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:15:24.878 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:15:25.417 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:15:25.417 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:15:25.418 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:15:25.418 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: seq 1 100 2024-05-15T13:15:25.419 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:25.419 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj1 = obj75 ']' 2024-05-15T13:15:25.419 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2024-05-15T13:15:25.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:25.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:25.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj2 = obj75 ']' 2024-05-15T13:15:25.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2024-05-15T13:15:25.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:25.576 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:25.576 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj3 = obj75 ']' 2024-05-15T13:15:25.576 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2024-05-15T13:15:25.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:25.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:25.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj4 = obj75 ']' 2024-05-15T13:15:25.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2024-05-15T13:15:25.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:25.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:25.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj5 = obj75 ']' 2024-05-15T13:15:25.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2024-05-15T13:15:25.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:25.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:25.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj6 = obj75 ']' 2024-05-15T13:15:25.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2024-05-15T13:15:25.698 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:25.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:25.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj7 = obj75 ']' 2024-05-15T13:15:25.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2024-05-15T13:15:25.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:25.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:25.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj8 = obj75 ']' 2024-05-15T13:15:25.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2024-05-15T13:15:25.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:25.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:25.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj9 = obj75 ']' 2024-05-15T13:15:25.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2024-05-15T13:15:25.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:25.806 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:25.806 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj10 = obj75 ']' 2024-05-15T13:15:25.807 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2024-05-15T13:15:25.835 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:25.836 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:25.836 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj11 = obj75 ']' 2024-05-15T13:15:25.836 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2024-05-15T13:15:25.864 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:25.865 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:25.865 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj12 = obj75 ']' 2024-05-15T13:15:25.866 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2024-05-15T13:15:25.894 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:25.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:25.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj13 = obj75 ']' 2024-05-15T13:15:25.895 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2024-05-15T13:15:25.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:25.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:25.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj14 = obj75 ']' 2024-05-15T13:15:25.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2024-05-15T13:15:25.984 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:25.985 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:25.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj15 = obj75 ']' 2024-05-15T13:15:25.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.014 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj16 = obj75 ']' 2024-05-15T13:15:26.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.067 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.068 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.068 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj17 = obj75 ']' 2024-05-15T13:15:26.068 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.096 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.097 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.097 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj18 = obj75 ']' 2024-05-15T13:15:26.097 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.127 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.127 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj19 = obj75 ']' 2024-05-15T13:15:26.127 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.157 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.157 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj20 = obj75 ']' 2024-05-15T13:15:26.157 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj21 = obj75 ']' 2024-05-15T13:15:26.186 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.215 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.216 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.216 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj22 = obj75 ']' 2024-05-15T13:15:26.216 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.245 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.245 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj23 = obj75 ']' 2024-05-15T13:15:26.245 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.274 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj24 = obj75 ']' 2024-05-15T13:15:26.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj25 = obj75 ']' 2024-05-15T13:15:26.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj26 = obj75 ']' 2024-05-15T13:15:26.334 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj27 = obj75 ']' 2024-05-15T13:15:26.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.392 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj28 = obj75 ']' 2024-05-15T13:15:26.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.421 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj29 = obj75 ']' 2024-05-15T13:15:26.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.452 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.452 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj30 = obj75 ']' 2024-05-15T13:15:26.452 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj31 = obj75 ']' 2024-05-15T13:15:26.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.510 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.511 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.511 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj32 = obj75 ']' 2024-05-15T13:15:26.511 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.587 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.588 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.588 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj33 = obj75 ']' 2024-05-15T13:15:26.588 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.698 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.698 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj34 = obj75 ']' 2024-05-15T13:15:26.698 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.735 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj35 = obj75 ']' 2024-05-15T13:15:26.736 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj36 = obj75 ']' 2024-05-15T13:15:26.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.793 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj37 = obj75 ']' 2024-05-15T13:15:26.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj38 = obj75 ']' 2024-05-15T13:15:26.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj39 = obj75 ']' 2024-05-15T13:15:26.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.880 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj40 = obj75 ']' 2024-05-15T13:15:26.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.910 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.911 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.911 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj41 = obj75 ']' 2024-05-15T13:15:26.911 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj42 = obj75 ']' 2024-05-15T13:15:26.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.968 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj43 = obj75 ']' 2024-05-15T13:15:26.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2024-05-15T13:15:26.998 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:26.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:26.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj44 = obj75 ']' 2024-05-15T13:15:26.999 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj45 = obj75 ']' 2024-05-15T13:15:27.028 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.056 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj46 = obj75 ']' 2024-05-15T13:15:27.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj47 = obj75 ']' 2024-05-15T13:15:27.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj48 = obj75 ']' 2024-05-15T13:15:27.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.145 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.146 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.146 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj49 = obj75 ']' 2024-05-15T13:15:27.146 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.174 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj50 = obj75 ']' 2024-05-15T13:15:27.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.203 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.204 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.204 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj51 = obj75 ']' 2024-05-15T13:15:27.204 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj52 = obj75 ']' 2024-05-15T13:15:27.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj53 = obj75 ']' 2024-05-15T13:15:27.263 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj54 = obj75 ']' 2024-05-15T13:15:27.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.321 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.321 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.321 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj55 = obj75 ']' 2024-05-15T13:15:27.322 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.351 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.351 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj56 = obj75 ']' 2024-05-15T13:15:27.351 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.379 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj57 = obj75 ']' 2024-05-15T13:15:27.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.408 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj58 = obj75 ']' 2024-05-15T13:15:27.409 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.437 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj59 = obj75 ']' 2024-05-15T13:15:27.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.467 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.467 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.468 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj60 = obj75 ']' 2024-05-15T13:15:27.468 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.495 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj61 = obj75 ']' 2024-05-15T13:15:27.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.524 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.525 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj62 = obj75 ']' 2024-05-15T13:15:27.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.553 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj63 = obj75 ']' 2024-05-15T13:15:27.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.583 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.583 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj64 = obj75 ']' 2024-05-15T13:15:27.583 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj65 = obj75 ']' 2024-05-15T13:15:27.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj66 = obj75 ']' 2024-05-15T13:15:27.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj67 = obj75 ']' 2024-05-15T13:15:27.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.699 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj68 = obj75 ']' 2024-05-15T13:15:27.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj69 = obj75 ']' 2024-05-15T13:15:27.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.785 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.786 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.786 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj70 = obj75 ']' 2024-05-15T13:15:27.787 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj71 = obj75 ']' 2024-05-15T13:15:27.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.844 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.845 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.845 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj72 = obj75 ']' 2024-05-15T13:15:27.845 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj73 = obj75 ']' 2024-05-15T13:15:27.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.903 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.904 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.904 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj74 = obj75 ']' 2024-05-15T13:15:27.904 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.948 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:27.949 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.949 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj75 = obj75 ']' 2024-05-15T13:15:27.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:684: TEST_ec_recovery_unfound: rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-05-15T13:15:27.975 INFO:tasks.workunit.client.0.smithi005.stderr:error getting pool-jerasure/obj75: (2) No such file or directory 2024-05-15T13:15:27.977 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:27.977 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj76 = obj75 ']' 2024-05-15T13:15:27.977 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.025 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.025 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj77 = obj75 ']' 2024-05-15T13:15:28.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj78 = obj75 ']' 2024-05-15T13:15:28.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj79 = obj75 ']' 2024-05-15T13:15:28.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj80 = obj75 ']' 2024-05-15T13:15:28.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.143 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.143 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.143 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj81 = obj75 ']' 2024-05-15T13:15:28.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj82 = obj75 ']' 2024-05-15T13:15:28.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.202 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.202 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj83 = obj75 ']' 2024-05-15T13:15:28.202 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.230 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.231 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.231 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj84 = obj75 ']' 2024-05-15T13:15:28.231 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj85 = obj75 ']' 2024-05-15T13:15:28.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj86 = obj75 ']' 2024-05-15T13:15:28.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj87 = obj75 ']' 2024-05-15T13:15:28.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.346 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj88 = obj75 ']' 2024-05-15T13:15:28.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.375 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.376 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.376 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj89 = obj75 ']' 2024-05-15T13:15:28.376 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.406 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj90 = obj75 ']' 2024-05-15T13:15:28.406 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.434 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj91 = obj75 ']' 2024-05-15T13:15:28.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj92 = obj75 ']' 2024-05-15T13:15:28.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.493 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj93 = obj75 ']' 2024-05-15T13:15:28.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.522 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj94 = obj75 ']' 2024-05-15T13:15:28.523 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.552 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.553 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.553 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj95 = obj75 ']' 2024-05-15T13:15:28.553 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.581 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj96 = obj75 ']' 2024-05-15T13:15:28.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.610 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj97 = obj75 ']' 2024-05-15T13:15:28.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj98 = obj75 ']' 2024-05-15T13:15:28.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj99 = obj75 ']' 2024-05-15T13:15:28.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.699 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.699 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-15T13:15:28.699 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj100 = obj75 ']' 2024-05-15T13:15:28.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2024-05-15T13:15:28.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:691: TEST_ec_recovery_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-15T13:15:28.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:693: TEST_ec_recovery_unfound: delete_erasure_coded_pool pool-jerasure 2024-05-15T13:15:28.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-15T13:15:28.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-15T13:15:29.370 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-jerasure' does not exist 2024-05-15T13:15:29.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-15T13:15:30.261 INFO:tasks.workunit.client.0.smithi005.stderr:erasure-code-profile myprofile does not exist 2024-05-15T13:15:30.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-15T13:15:30.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-15T13:15:30.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T13:15:30.279 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-15T13:15:30.280 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:15:30.280 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:15:30.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:15:30.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:15:30.280 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:15:30.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:15:30.395 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T13:15:30.395 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T13:15:30.396 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T13:15:30.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T13:15:30.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T13:15:30.397 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T13:15:30.398 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:15:30.398 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T13:15:30.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T13:15:30.399 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:15:30.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:15:30.400 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T13:15:30.401 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-15T13:15:30.443 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T13:15:30.443 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:15:30.443 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:15:30.443 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80702 2024-05-15T13:15:30.444 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T13:15:30.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T13:15:30.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-15T13:15:30.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-15T13:15:30.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-15T13:15:30.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-15T13:15:30.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-15T13:15:30.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T13:15:30.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-15T13:15:30.445 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:15:30.445 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:15:30.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:15:30.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:15:30.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:15:30.446 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:15:30.447 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T13:15:30.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T13:15:30.448 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T13:15:30.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T13:15:30.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T13:15:30.449 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T13:15:30.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:15:30.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T13:15:30.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T13:15:30.451 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:15:30.451 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:15:30.452 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T13:15:30.453 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-15T13:15:30.455 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T13:15:30.455 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:15:30.455 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:15:30.455 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80702 2024-05-15T13:15:30.455 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T13:15:30.455 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T13:15:30.456 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-15T13:15:30.456 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T13:15:30.456 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:15:30.456 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:15:30.456 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.80702 2024-05-15T13:15:30.457 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T13:15:30.458 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-15T13:15:30.458 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T13:15:30.458 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-15T13:15:30.458 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-15T13:15:30.459 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-15T13:15:30.459 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T13:15:30.459 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T13:15:30.459 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T13:15:30.459 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-15T13:15:30.459 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-15T13:15:30.840 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T13:15:30.841 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:15:30.841 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:15:30.841 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:15:30.842 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:15:30.842 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:15:30.842 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:15:30.842 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-15T13:15:31.060 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T13:15:31.064 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T13:15:31.064 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T13:15:31.064 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T13:15:31.064 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T13:15:31.065 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T13:15:31.065 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T13:15:31.065 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T13:15:31.065 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T13:15:31.065 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:15:31.065 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:15:31.065 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:15:31.066 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T13:15:31.066 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T13:15:31.066 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.80702/ceph-mon.a.asok config get fsid 2024-05-15T13:15:31.183 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T13:15:31.184 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T13:15:31.184 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T13:15:31.184 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T13:15:31.184 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T13:15:31.184 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T13:15:31.184 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T13:15:31.184 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T13:15:31.185 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:15:31.185 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:15:31.185 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:15:31.185 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T13:15:31.185 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T13:15:31.186 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.80702/ceph-mon.a.asok config get mon_host 2024-05-15T13:15:31.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-15T13:15:31.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-15T13:15:31.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T13:15:31.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T13:15:31.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T13:15:31.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-15T13:15:31.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T13:15:32.195 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T13:15:32.195 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:15:32.196 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:15:32.196 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:15:32.196 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:15:32.196 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:15:32.197 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:15:32.198 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T13:15:32.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T13:15:32.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-15T13:15:32.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-15T13:15:32.563 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd' already exists 2024-05-15T13:15:32.581 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T13:15:33.584 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-15T13:15:33.584 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T13:15:33.584 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T13:15:33.585 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:15:33.585 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:15:33.585 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:15:33.585 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T13:15:33.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-15T13:15:33.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-mon.a.asok log flush 2024-05-15T13:15:33.686 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T13:15:33.699 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-15T13:15:33.702 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T13:15:30.892+0000 7f630ef72c80 0 load: jerasure load: lrc 2024-05-15T13:15:33.702 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_single_recovery_error td/test-erasure-eio 2024-05-15T13:15:33.702 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:390: TEST_ec_single_recovery_error: local dir=td/test-erasure-eio 2024-05-15T13:15:33.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:391: TEST_ec_single_recovery_error: local objname=myobject 2024-05-15T13:15:33.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:393: TEST_ec_single_recovery_error: setup_osds 7 2024-05-15T13:15:33.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-05-15T13:15:33.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-15T13:15:33.703 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-05-15T13:15:33.705 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-05-15T13:15:33.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:15:33.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-15T13:15:33.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:15:33.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:15:33.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T13:15:33.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:15:33.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-15T13:15:33.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:15:33.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:15:33.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:15:33.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:15:33.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-15T13:15:33.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-15T13:15:33.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:15:33.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:15:33.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:15:33.709 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:15:33.709 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:15:33.709 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:15:33.710 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:15:33.710 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:15:33.710 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:15:33.710 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:15:33.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:15:33.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:15:33.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:15:33.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:15:33.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:15:33.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:15:33.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:15:33.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:15:33.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:15:33.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:15:33.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:15:33.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:15:33.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-15T13:15:33.711 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:15:33.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c9c13d71-0a76-4ddc-bda1-75cc5d391055 2024-05-15T13:15:33.712 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 c9c13d71-0a76-4ddc-bda1-75cc5d391055 2024-05-15T13:15:33.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c9c13d71-0a76-4ddc-bda1-75cc5d391055' 2024-05-15T13:15:33.712 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:15:33.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB1tURmr74HLBAA2ziVOm+BZ7DBI+BwbwiZgA== 2024-05-15T13:15:33.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB1tURmr74HLBAA2ziVOm+BZ7DBI+BwbwiZgA=="}' 2024-05-15T13:15:33.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c9c13d71-0a76-4ddc-bda1-75cc5d391055 -i td/test-erasure-eio/0/new.json 2024-05-15T13:15:34.048 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:15:34.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-15T13:15:34.064 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB1tURmr74HLBAA2ziVOm+BZ7DBI+BwbwiZgA== --osd-uuid c9c13d71-0a76-4ddc-bda1-75cc5d391055 2024-05-15T13:15:34.099 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:15:34.104+0000 7f7eb379e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:15:34.116 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:15:34.120+0000 7f7eb379e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:15:34.118 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:15:34.124+0000 7f7eb379e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:15:34.120 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:15:34.124+0000 7f7eb379e7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T13:15:34.120 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:15:34.124+0000 7f7eb379e7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T13:15:34.120 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:15:34.124+0000 7f7eb379e7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T13:15:34.279 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:15:34.284+0000 7f7eb379e7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-15T13:15:39.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-15T13:15:39.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:15:39.925 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T13:15:39.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T13:15:39.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:15:40.505 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T13:15:40.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T13:15:40.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:15:40.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:15:40.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:15:40.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:15:40.538 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:15:40.544+0000 7f23002227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:15:40.540 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:15:40.544+0000 7f23002227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:15:40.542 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:15:40.548+0000 7f23002227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:15:40.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T13:15:40.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:15:40.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T13:15:40.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:15:40.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:15:40.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:15:40.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:15:40.919 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:15:40.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:15:40.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:15:41.123 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:15:41.127+0000 7f23002227c0 -1 Falling back to public interface 2024-05-15T13:15:41.387 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:15:42.389 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:15:42.389 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:15:42.390 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:15:42.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:15:42.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:15:42.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:15:42.807 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:15:43.654 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:15:43.659+0000 7f23002227c0 -1 osd.0 0 log_to_monitors true 2024-05-15T13:15:43.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:15:43.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:15:43.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:15:43.810 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:15:43.810 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:15:43.810 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:15:44.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:15:45.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:15:45.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:15:45.366 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:15:45.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:15:45.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:15:45.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:15:45.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:15:46.801 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:15:46.802 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:15:46.802 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:15:46.802 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:15:46.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:15:46.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:15:47.395 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:15:48.396 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:15:48.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:15:48.397 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:15:48.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:15:48.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:15:48.398 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:15:48.838 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:15:49.840 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:15:49.840 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:15:49.840 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:15:49.841 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:15:49.841 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:15:49.841 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:15:50.372 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:15:51.373 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:15:51.373 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:15:51.374 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:15:51.374 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:15:51.374 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:15:51.374 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:15:51.806 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:15:52.808 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:15:52.808 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:15:52.808 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:15:52.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:15:52.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:15:52.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:15:53.341 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:15:54.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:15:54.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:15:54.343 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:15:54.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:15:54.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:15:54.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:15:54.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:15:55.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:15:55.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:15:55.964 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:15:55.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:15:55.965 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:15:55.965 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:15:56.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:15:57.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:15:57.392 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:15:57.392 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:15:57.392 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:15:57.392 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:15:57.392 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:15:58.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:15:59.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:15:59.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:15:59.007 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:15:59.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:15:59.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:15:59.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:15:59.915 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:00.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:00.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:00.918 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:16:00.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:16:00.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:00.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:16:01.785 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:02.787 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:02.787 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:02.787 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:16:02.788 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:16:02.788 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:02.788 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:16:03.368 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:04.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:04.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:04.370 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:16:04.371 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:16:04.371 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:04.371 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:16:05.166 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3818895919,v1:127.0.0.1:6803/3818895919] [v2:127.0.0.1:6804/3818895919,v1:127.0.0.1:6805/3818895919] exists,up c9c13d71-0a76-4ddc-bda1-75cc5d391055 2024-05-15T13:16:05.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:16:05.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:16:05.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:16:05.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:16:05.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-15T13:16:05.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:16:05.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:16:05.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-15T13:16:05.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:16:05.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-15T13:16:05.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:16:05.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:16:05.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:16:05.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:16:05.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-15T13:16:05.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-15T13:16:05.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:16:05.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:16:05.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:16:05.172 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:16:05.172 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:16:05.172 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:16:05.173 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:16:05.173 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:16:05.173 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:16:05.173 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:16:05.174 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:16:05.174 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:16:05.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:16:05.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:16:05.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:16:05.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:16:05.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:16:05.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:16:05.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:16:05.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:16:05.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:16:05.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:16:05.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-15T13:16:05.175 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:16:05.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a9a340ec-c5e9-498e-80ae-6a6ee1a7ac6c 2024-05-15T13:16:05.176 INFO:tasks.workunit.client.0.smithi005.stdout:add osd1 a9a340ec-c5e9-498e-80ae-6a6ee1a7ac6c 2024-05-15T13:16:05.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a9a340ec-c5e9-498e-80ae-6a6ee1a7ac6c' 2024-05-15T13:16:05.176 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:16:05.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCVtURmVfceDBAAlgBU0FTundI6YT39/aQfFw== 2024-05-15T13:16:05.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCVtURmVfceDBAAlgBU0FTundI6YT39/aQfFw=="}' 2024-05-15T13:16:05.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a9a340ec-c5e9-498e-80ae-6a6ee1a7ac6c -i td/test-erasure-eio/1/new.json 2024-05-15T13:16:05.883 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:16:05.901 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-15T13:16:05.902 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCVtURmVfceDBAAlgBU0FTundI6YT39/aQfFw== --osd-uuid a9a340ec-c5e9-498e-80ae-6a6ee1a7ac6c 2024-05-15T13:16:05.931 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:05.935+0000 7fdfb28cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:16:05.936 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:05.939+0000 7fdfb28cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:16:05.937 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:05.943+0000 7fdfb28cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:16:05.938 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:05.943+0000 7fdfb28cf7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T13:16:05.938 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:05.943+0000 7fdfb28cf7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T13:16:05.938 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:05.943+0000 7fdfb28cf7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T13:16:05.976 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:05.979+0000 7fdfb28cf7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-15T13:16:11.787 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-15T13:16:11.787 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:16:11.788 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd1 key to auth repository 2024-05-15T13:16:11.788 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-15T13:16:11.789 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:16:12.349 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-15T13:16:12.349 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.1 2024-05-15T13:16:12.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:16:12.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:16:12.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:16:12.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:16:12.381 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:12.383+0000 7f428af2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:16:12.406 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:12.411+0000 7f428af2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:16:12.407 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:12.411+0000 7f428af2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:16:12.838 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-15T13:16:12.839 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:16:12.839 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-15T13:16:12.839 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:16:12.839 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:16:12.839 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:12.839 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:16:12.839 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:16:12.839 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:12.839 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:16:12.967 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:12.971+0000 7f428af2d7c0 -1 Falling back to public interface 2024-05-15T13:16:13.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:14.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:14.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:14.259 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:16:14.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:16:14.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:14.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:16:14.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:15.549 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:15.551+0000 7f428af2d7c0 -1 osd.1 0 log_to_monitors true 2024-05-15T13:16:15.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:15.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:15.672 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:16:15.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:16:15.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:15.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:16:16.296 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:17.298 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:17.298 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:17.299 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:16:17.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:16:17.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:17.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:16:17.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:18.825 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:18.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:18.826 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:16:18.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:16:18.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:18.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:16:19.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:20.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:20.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:20.367 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:16:20.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:16:20.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:20.367 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:16:20.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:21.799 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:21.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:21.800 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:16:21.801 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:16:21.801 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:21.801 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:16:22.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:23.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:23.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:23.314 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:16:23.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:16:23.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:23.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:16:23.818 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:24.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:24.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:24.820 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:16:24.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:16:24.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:24.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:16:25.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:26.383 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:26.383 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:26.383 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:16:26.383 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:16:26.384 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:26.384 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:16:26.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:27.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:27.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:27.871 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:16:27.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:16:27.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:27.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:16:28.963 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:29.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:29.965 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:29.965 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:16:29.965 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:16:29.965 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:29.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:16:30.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:31.947 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:31.947 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:31.948 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:16:31.948 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:16:31.948 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:31.948 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:16:32.693 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:33.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:33.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:33.695 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:16:33.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:16:33.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:33.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:16:34.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:35.732 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:35.732 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:35.732 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:16:35.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:16:35.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:35.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:16:36.597 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:37.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:37.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:37.599 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:16:37.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:16:37.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:37.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:16:38.175 INFO:tasks.workunit.client.0.smithi005.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/309502239,v1:127.0.0.1:6811/309502239] [v2:127.0.0.1:6812/309502239,v1:127.0.0.1:6813/309502239] exists,up a9a340ec-c5e9-498e-80ae-6a6ee1a7ac6c 2024-05-15T13:16:38.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:16:38.178 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:16:38.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:16:38.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:16:38.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-15T13:16:38.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:16:38.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:16:38.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-15T13:16:38.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:16:38.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-15T13:16:38.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:16:38.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:16:38.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:16:38.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:16:38.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-15T13:16:38.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-15T13:16:38.179 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:16:38.180 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:16:38.180 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:16:38.180 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:16:38.180 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:16:38.180 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:16:38.180 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:16:38.180 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:16:38.180 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:16:38.180 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:16:38.180 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:16:38.180 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:16:38.180 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:16:38.180 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:16:38.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:16:38.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:16:38.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:16:38.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:16:38.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:16:38.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:16:38.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:16:38.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:16:38.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-15T13:16:38.182 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:16:38.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=548c8577-7a20-4cf4-8afe-916f1bd906b7 2024-05-15T13:16:38.182 INFO:tasks.workunit.client.0.smithi005.stdout:add osd2 548c8577-7a20-4cf4-8afe-916f1bd906b7 2024-05-15T13:16:38.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 548c8577-7a20-4cf4-8afe-916f1bd906b7' 2024-05-15T13:16:38.182 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:16:38.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC2tURmJNUpDBAAmHI4yGJNHFR8ylDDyjoAIQ== 2024-05-15T13:16:38.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC2tURmJNUpDBAAmHI4yGJNHFR8ylDDyjoAIQ=="}' 2024-05-15T13:16:38.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 548c8577-7a20-4cf4-8afe-916f1bd906b7 -i td/test-erasure-eio/2/new.json 2024-05-15T13:16:38.724 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:16:38.740 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-15T13:16:38.741 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC2tURmJNUpDBAAmHI4yGJNHFR8ylDDyjoAIQ== --osd-uuid 548c8577-7a20-4cf4-8afe-916f1bd906b7 2024-05-15T13:16:38.777 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:38.778+0000 7f147be3c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:16:38.779 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:38.782+0000 7f147be3c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:16:38.780 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:38.782+0000 7f147be3c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:16:38.781 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:38.782+0000 7f147be3c7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T13:16:38.781 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:38.782+0000 7f147be3c7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T13:16:38.781 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:38.782+0000 7f147be3c7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T13:16:38.824 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:38.826+0000 7f147be3c7c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-15T13:16:44.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-15T13:16:44.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:16:44.713 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-15T13:16:44.713 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd2 key to auth repository 2024-05-15T13:16:44.713 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:16:45.287 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.2 2024-05-15T13:16:45.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-15T13:16:45.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:16:45.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:16:45.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:16:45.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:16:45.321 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:45.322+0000 7fe58f0f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:16:45.331 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:45.334+0000 7fe58f0f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:16:45.332 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:45.334+0000 7fe58f0f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:16:45.699 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-15T13:16:45.699 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:16:45.699 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-15T13:16:45.699 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:16:45.699 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:16:45.699 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:45.699 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:16:45.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:16:45.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:45.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:16:45.908 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:45.910+0000 7fe58f0f27c0 -1 Falling back to public interface 2024-05-15T13:16:46.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:47.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:47.125 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:47.125 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:16:47.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:16:47.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:47.126 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:16:47.563 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:48.381 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:16:48.382+0000 7fe58f0f27c0 -1 osd.2 0 log_to_monitors true 2024-05-15T13:16:48.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:48.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:48.565 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:16:48.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:16:48.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:48.565 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:16:49.017 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:50.019 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:50.019 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:50.020 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:16:50.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:16:50.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:50.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:16:50.496 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:51.498 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:51.498 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:51.499 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:16:51.499 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:16:51.499 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:51.499 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:16:51.968 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:52.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:52.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:52.971 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:16:52.971 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:16:52.971 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:52.971 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:16:53.416 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:54.418 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:54.419 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:54.419 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:16:54.419 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:16:54.420 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:54.420 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:16:54.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:55.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:55.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:55.860 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:16:55.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:16:55.860 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:55.861 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:16:56.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:57.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:57.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:57.314 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:16:57.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:16:57.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:57.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:16:57.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:16:58.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:16:58.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:16:58.765 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:16:58.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:16:58.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:16:58.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:16:59.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:00.316 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:00.316 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:00.316 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:17:00.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:17:00.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:00.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:17:01.013 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:02.015 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:02.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:02.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:17:02.016 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:17:02.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:02.016 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:17:02.848 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:03.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:03.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:03.850 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:17:03.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:17:03.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:03.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:17:04.864 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:05.866 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:05.867 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:05.867 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:17:05.867 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:17:05.867 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:05.867 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:17:06.683 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:07.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:07.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:07.686 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:17:07.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:17:07.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:07.686 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:17:08.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:09.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:09.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:09.538 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:17:09.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:17:09.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:09.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:17:10.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:11.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:11.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:11.614 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:17:11.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:17:11.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:11.614 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:17:12.256 INFO:tasks.workunit.client.0.smithi005.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2914599755,v1:127.0.0.1:6819/2914599755] [v2:127.0.0.1:6820/2914599755,v1:127.0.0.1:6821/2914599755] exists,up 548c8577-7a20-4cf4-8afe-916f1bd906b7 2024-05-15T13:17:12.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:17:12.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:17:12.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:17:12.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:17:12.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-15T13:17:12.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:17:12.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:17:12.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-15T13:17:12.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:17:12.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-15T13:17:12.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:17:12.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:17:12.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:17:12.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:17:12.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-15T13:17:12.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-15T13:17:12.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:17:12.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:17:12.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:17:12.259 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:17:12.259 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:17:12.259 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:17:12.260 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:17:12.260 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:17:12.260 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:17:12.260 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:17:12.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:17:12.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:17:12.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:17:12.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:17:12.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:17:12.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:17:12.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:17:12.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:17:12.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:17:12.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:17:12.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:17:12.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:17:12.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-15T13:17:12.262 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:17:12.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f1593aae-7254-410d-940e-f0459196cccd 2024-05-15T13:17:12.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 f1593aae-7254-410d-940e-f0459196cccd' 2024-05-15T13:17:12.262 INFO:tasks.workunit.client.0.smithi005.stdout:add osd3 f1593aae-7254-410d-940e-f0459196cccd 2024-05-15T13:17:12.263 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:17:12.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDYtURmha90ERAAighEZatwTCv4I2apn9mqgA== 2024-05-15T13:17:12.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDYtURmha90ERAAighEZatwTCv4I2apn9mqgA=="}' 2024-05-15T13:17:12.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f1593aae-7254-410d-940e-f0459196cccd -i td/test-erasure-eio/3/new.json 2024-05-15T13:17:12.955 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:17:12.977 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-15T13:17:12.979 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDYtURmha90ERAAighEZatwTCv4I2apn9mqgA== --osd-uuid f1593aae-7254-410d-940e-f0459196cccd 2024-05-15T13:17:13.017 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:13.018+0000 7f3165e5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:17:13.036 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:13.038+0000 7f3165e5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:17:13.037 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:13.038+0000 7f3165e5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:17:13.038 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:13.038+0000 7f3165e5a7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T13:17:13.038 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:13.038+0000 7f3165e5a7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T13:17:13.038 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:13.038+0000 7f3165e5a7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T13:17:13.145 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:13.146+0000 7f3165e5a7c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-15T13:17:19.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-15T13:17:19.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:17:19.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-15T13:17:19.273 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd3 key to auth repository 2024-05-15T13:17:19.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:17:19.824 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.3 2024-05-15T13:17:19.825 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-15T13:17:19.825 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:17:19.825 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:17:19.825 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:17:19.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:17:19.857 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:19.858+0000 7f4e94a307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:17:19.860 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:19.862+0000 7f4e94a307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:17:19.861 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:19.862+0000 7f4e94a307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:17:20.245 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-15T13:17:20.245 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:17:20.245 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-15T13:17:20.245 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:17:20.245 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:17:20.245 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:20.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:17:20.246 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:17:20.249 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:20.249 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:17:20.444 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:20.446+0000 7f4e94a307c0 -1 Falling back to public interface 2024-05-15T13:17:20.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:21.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:21.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:21.673 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:17:21.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:17:21.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:21.674 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:17:22.142 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:22.993 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:22.993+0000 7f4e94a307c0 -1 osd.3 0 log_to_monitors true 2024-05-15T13:17:23.143 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:23.143 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:23.143 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:17:23.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:17:23.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:23.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:17:23.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:24.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:24.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:24.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:17:24.661 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:17:24.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:24.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:17:25.154 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:26.156 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:26.156 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:26.156 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:17:26.157 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:17:26.157 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:26.157 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:17:26.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:27.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:27.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:27.584 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:17:27.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:17:27.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:27.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:17:28.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:29.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:29.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:29.081 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:17:29.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:17:29.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:29.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:17:29.583 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:30.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:30.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:30.585 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:17:30.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:17:30.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:30.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:17:31.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:32.046 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:32.046 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:32.046 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:17:32.047 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:17:32.047 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:32.047 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:17:32.524 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:33.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:33.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:33.526 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:17:33.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:17:33.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:33.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:17:34.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:35.041 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:35.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:35.042 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:17:35.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:17:35.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:35.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:17:35.524 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:36.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:36.526 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:36.526 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:17:36.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:17:36.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:36.527 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:17:37.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:38.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:38.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:38.288 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:17:38.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:17:38.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:38.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:17:39.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:40.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:40.118 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:40.118 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:17:40.119 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:17:40.119 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:40.119 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:17:40.833 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:41.835 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:41.836 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:41.836 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:17:41.836 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:17:41.836 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:41.836 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:17:42.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:43.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:43.816 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:43.817 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:17:43.817 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:17:43.817 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:43.817 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:17:44.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:45.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:45.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:45.313 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:17:45.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:17:45.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:45.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:17:46.191 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:47.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:47.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:47.193 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:17:47.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:17:47.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:47.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:17:47.625 INFO:tasks.workunit.client.0.smithi005.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1221950755,v1:127.0.0.1:6827/1221950755] [v2:127.0.0.1:6828/1221950755,v1:127.0.0.1:6829/1221950755] exists,up f1593aae-7254-410d-940e-f0459196cccd 2024-05-15T13:17:47.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:17:47.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:17:47.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:17:47.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:17:47.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-05-15T13:17:47.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:17:47.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:17:47.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-05-15T13:17:47.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:17:47.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-05-15T13:17:47.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:17:47.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:17:47.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:17:47.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:17:47.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-05-15T13:17:47.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-05-15T13:17:47.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:17:47.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:17:47.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:17:47.628 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:17:47.628 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:17:47.628 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:17:47.630 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:17:47.630 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:17:47.630 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:17:47.630 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:17:47.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:17:47.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:17:47.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:17:47.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:17:47.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:17:47.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:17:47.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:17:47.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:17:47.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:17:47.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:17:47.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:17:47.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:17:47.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-05-15T13:17:47.650 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:17:47.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=29a6ab03-643a-45c0-90f8-cb2c2d941dd1 2024-05-15T13:17:47.651 INFO:tasks.workunit.client.0.smithi005.stdout:add osd4 29a6ab03-643a-45c0-90f8-cb2c2d941dd1 2024-05-15T13:17:47.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 29a6ab03-643a-45c0-90f8-cb2c2d941dd1' 2024-05-15T13:17:47.652 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:17:47.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7tURmu6GBKBAAhnN4NRNaiKCx5CgAnPI+4w== 2024-05-15T13:17:47.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7tURmu6GBKBAAhnN4NRNaiKCx5CgAnPI+4w=="}' 2024-05-15T13:17:47.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 29a6ab03-643a-45c0-90f8-cb2c2d941dd1 -i td/test-erasure-eio/4/new.json 2024-05-15T13:17:48.516 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:17:48.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-05-15T13:17:48.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7tURmu6GBKBAAhnN4NRNaiKCx5CgAnPI+4w== --osd-uuid 29a6ab03-643a-45c0-90f8-cb2c2d941dd1 2024-05-15T13:17:48.581 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:48.581+0000 7fa607f627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:17:48.584 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:48.585+0000 7fa607f627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:17:48.585 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:48.585+0000 7fa607f627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:17:48.586 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:48.585+0000 7fa607f627c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T13:17:48.586 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:48.585+0000 7fa607f627c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T13:17:48.586 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:48.585+0000 7fa607f627c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-15T13:17:48.733 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:48.733+0000 7fa607f627c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-05-15T13:17:55.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-05-15T13:17:55.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:17:55.269 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd4 key to auth repository 2024-05-15T13:17:55.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-05-15T13:17:55.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:17:55.881 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.4 2024-05-15T13:17:55.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-05-15T13:17:55.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:17:55.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:17:55.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:17:55.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:17:55.914 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:55.913+0000 7f7ddde777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:17:55.935 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:55.937+0000 7f7ddde777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:17:55.937 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:55.937+0000 7f7ddde777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:17:56.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-05-15T13:17:56.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:17:56.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-05-15T13:17:56.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:17:56.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:17:56.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:56.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:17:56.361 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:17:56.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:56.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:17:56.496 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:56.497+0000 7f7ddde777c0 -1 Falling back to public interface 2024-05-15T13:17:56.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:57.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:57.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:57.823 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:17:57.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:17:57.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:57.824 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:17:58.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:17:58.994 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:17:58.993+0000 7f7ddde777c0 -1 osd.4 0 log_to_monitors true 2024-05-15T13:17:59.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:17:59.267 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:17:59.268 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:17:59.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:17:59.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:17:59.268 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:17:59.798 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:00.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:00.801 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:00.801 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:18:00.801 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:18:00.801 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:00.801 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:18:01.404 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:02.406 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:02.407 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:02.407 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:18:02.407 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:18:02.407 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:02.407 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:18:02.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:03.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:03.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:03.876 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:18:03.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:18:03.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:03.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:18:04.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:05.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:05.309 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:05.310 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:18:05.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:18:05.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:05.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:18:05.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:06.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:06.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:06.733 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:18:06.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:18:06.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:06.734 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:18:07.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:08.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:08.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:08.196 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:18:08.196 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:18:08.196 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:08.196 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:18:08.742 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:09.744 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:09.744 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:09.744 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:18:09.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:18:09.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:09.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:18:10.197 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:11.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:11.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:11.199 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:18:11.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:18:11.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:11.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:18:12.130 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:13.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:13.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:13.133 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:18:13.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:18:13.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:13.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:18:13.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:14.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:14.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:14.796 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:18:14.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:18:14.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:14.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:18:15.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:16.274 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:16.274 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:16.274 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:18:16.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:18:16.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:16.275 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:18:16.964 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:17.966 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:17.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:17.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:18:17.967 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:18:17.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:17.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:18:18.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:19.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:19.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:19.920 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:18:19.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:18:19.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:19.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:18:20.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:21.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:21.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:21.380 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:18:21.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:18:21.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:21.381 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:18:22.029 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:23.030 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:23.031 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:23.031 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:18:23.031 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:18:23.031 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:23.032 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-15T13:18:23.678 INFO:tasks.workunit.client.0.smithi005.stdout:osd.4 up in weight 1 up_from 38 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2450880111,v1:127.0.0.1:6835/2450880111] [v2:127.0.0.1:6836/2450880111,v1:127.0.0.1:6837/2450880111] exists,up 29a6ab03-643a-45c0-90f8-cb2c2d941dd1 2024-05-15T13:18:23.680 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:18:23.680 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:18:23.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:18:23.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:18:23.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-05-15T13:18:23.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:18:23.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:18:23.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-05-15T13:18:23.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:18:23.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-05-15T13:18:23.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:18:23.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:18:23.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:18:23.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:18:23.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-05-15T13:18:23.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-05-15T13:18:23.681 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:18:23.682 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:18:23.682 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:18:23.682 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:18:23.682 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:18:23.682 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:18:23.683 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:18:23.683 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:18:23.683 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:18:23.684 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:18:23.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:18:23.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:18:23.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:18:23.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:18:23.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:18:23.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:18:23.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:18:23.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:18:23.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:18:23.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:18:23.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:18:23.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:18:23.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-05-15T13:18:23.685 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:18:23.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5170bfdb-7052-4ca8-aaf1-4b26bee0186a 2024-05-15T13:18:23.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 5170bfdb-7052-4ca8-aaf1-4b26bee0186a' 2024-05-15T13:18:23.685 INFO:tasks.workunit.client.0.smithi005.stdout:add osd5 5170bfdb-7052-4ca8-aaf1-4b26bee0186a 2024-05-15T13:18:23.686 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:18:23.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAftkRmESp4KhAAynYEgHcHw/IV9S+4XyW57A== 2024-05-15T13:18:23.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAftkRmESp4KhAAynYEgHcHw/IV9S+4XyW57A=="}' 2024-05-15T13:18:23.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5170bfdb-7052-4ca8-aaf1-4b26bee0186a -i td/test-erasure-eio/5/new.json 2024-05-15T13:18:24.436 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:18:24.453 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-05-15T13:18:24.455 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAftkRmESp4KhAAynYEgHcHw/IV9S+4XyW57A== --osd-uuid 5170bfdb-7052-4ca8-aaf1-4b26bee0186a 2024-05-15T13:18:24.490 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:18:24.488+0000 7f911228f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:18:24.492 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:18:24.492+0000 7f911228f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:18:24.494 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:18:24.492+0000 7f911228f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:18:24.494 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:18:24.492+0000 7f911228f7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-15T13:18:24.494 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:18:24.492+0000 7f911228f7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-15T13:18:24.494 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:18:24.492+0000 7f911228f7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-15T13:18:24.607 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:18:24.604+0000 7f911228f7c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-05-15T13:18:31.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-05-15T13:18:31.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:18:31.506 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd5 key to auth repository 2024-05-15T13:18:31.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-05-15T13:18:31.507 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:18:32.080 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-05-15T13:18:32.081 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.5 2024-05-15T13:18:32.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:18:32.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:18:32.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:18:32.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:18:32.114 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:18:32.112+0000 7f36daef67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:18:32.135 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:18:32.136+0000 7f36daef67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:18:32.136 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:18:32.136+0000 7f36daef67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:18:32.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-05-15T13:18:32.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:18:32.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-05-15T13:18:32.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:18:32.549 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:18:32.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:18:32.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:32.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:18:32.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:32.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:18:32.708 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:18:32.708+0000 7f36daef67c0 -1 Falling back to public interface 2024-05-15T13:18:33.000 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:34.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:34.001 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:34.001 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:18:34.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:18:34.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:34.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:18:34.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:35.389 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:18:35.388+0000 7f36daef67c0 -1 osd.5 0 log_to_monitors true 2024-05-15T13:18:35.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:35.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:35.429 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:18:35.430 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:18:35.430 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:35.430 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:18:35.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:36.936 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:36.936 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:36.937 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:18:36.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:18:36.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:36.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:18:37.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:38.663 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:38.663 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:38.663 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:18:38.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:18:38.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:38.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:18:39.397 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:40.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:40.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:40.399 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:18:40.400 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:18:40.400 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:40.400 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:18:40.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:41.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:41.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:41.821 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:18:41.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:18:41.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:41.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:18:42.368 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:43.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:43.370 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:43.370 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:18:43.371 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:18:43.371 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:43.371 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:18:43.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:44.810 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:44.811 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:44.811 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:18:44.811 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:18:44.811 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:44.811 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:18:45.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:46.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:46.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:46.358 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:18:46.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:18:46.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:46.359 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:18:46.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:47.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:47.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:47.821 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:18:47.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:18:47.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:47.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:18:48.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:49.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:49.349 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:49.349 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:18:49.349 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:18:49.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:49.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:18:49.885 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:50.887 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:50.887 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:50.887 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:18:50.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:18:50.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:50.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:18:51.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:52.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:52.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:52.923 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:18:52.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:18:52.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:52.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:18:53.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:54.642 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:54.642 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:54.643 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:18:54.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:18:54.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:54.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:18:55.510 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:56.512 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:56.512 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:56.513 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:18:56.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:18:56.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:56.513 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:18:57.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:18:58.371 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:18:58.371 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:18:58.371 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:18:58.371 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:18:58.371 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:18:58.372 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:18:59.230 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:19:00.232 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:19:00.232 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:00.232 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:19:00.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:19:00.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:00.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:19:01.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:19:02.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:19:02.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:02.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:19:02.100 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:19:02.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:02.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-15T13:19:02.585 INFO:tasks.workunit.client.0.smithi005.stdout:osd.5 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2448048742,v1:127.0.0.1:6843/2448048742] [v2:127.0.0.1:6844/2448048742,v1:127.0.0.1:6845/2448048742] exists,up 5170bfdb-7052-4ca8-aaf1-4b26bee0186a 2024-05-15T13:19:02.588 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:19:02.588 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:19:02.588 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:19:02.588 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:19:02.588 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-05-15T13:19:02.588 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:19:02.589 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:19:02.589 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-05-15T13:19:02.589 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:19:02.589 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-05-15T13:19:02.589 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:19:02.589 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:19:02.589 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:19:02.589 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:19:02.589 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-05-15T13:19:02.589 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-05-15T13:19:02.589 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:19:02.589 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:19:02.589 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:19:02.590 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:19:02.590 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:19:02.590 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:19:02.590 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:19:02.590 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:19:02.590 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:19:02.590 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:19:02.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:19:02.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:19:02.590 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:19:02.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:19:02.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:19:02.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:19:02.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:19:02.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:19:02.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:19:02.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:19:02.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:19:02.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:19:02.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-05-15T13:19:02.592 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:19:02.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=063e2e4e-c1a7-40f9-bcd6-f71d6d377737 2024-05-15T13:19:02.592 INFO:tasks.workunit.client.0.smithi005.stdout:add osd6 063e2e4e-c1a7-40f9-bcd6-f71d6d377737 2024-05-15T13:19:02.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 063e2e4e-c1a7-40f9-bcd6-f71d6d377737' 2024-05-15T13:19:02.592 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:19:02.610 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGtkRmtQCMJBAAGhh56XO79t9bcT1ELhUQPg== 2024-05-15T13:19:02.610 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGtkRmtQCMJBAAGhh56XO79t9bcT1ELhUQPg=="}' 2024-05-15T13:19:02.610 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 063e2e4e-c1a7-40f9-bcd6-f71d6d377737 -i td/test-erasure-eio/6/new.json 2024-05-15T13:19:03.104 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:19:03.122 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-05-15T13:19:03.124 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGtkRmtQCMJBAAGhh56XO79t9bcT1ELhUQPg== --osd-uuid 063e2e4e-c1a7-40f9-bcd6-f71d6d377737 2024-05-15T13:19:03.154 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:19:03.156+0000 7fba577b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:19:03.156 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:19:03.160+0000 7fba577b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:19:03.157 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:19:03.160+0000 7fba577b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:19:03.158 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:19:03.160+0000 7fba577b57c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-15T13:19:03.158 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:19:03.160+0000 7fba577b57c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-15T13:19:03.158 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:19:03.160+0000 7fba577b57c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-15T13:19:03.234 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:19:03.236+0000 7fba577b57c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-05-15T13:19:11.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-05-15T13:19:11.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:19:11.558 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd6 key to auth repository 2024-05-15T13:19:11.559 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-05-15T13:19:11.559 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:19:12.115 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.6 2024-05-15T13:19:12.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-05-15T13:19:12.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:19:12.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:19:12.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:19:12.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:19:12.148 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:19:12.151+0000 7f507149a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:19:12.180 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:19:12.183+0000 7f507149a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:19:12.182 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:19:12.183+0000 7f507149a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:19:12.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-05-15T13:19:12.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:19:12.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-05-15T13:19:12.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:19:12.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:19:12.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:12.558 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:19:12.558 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:19:12.559 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:12.559 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:19:12.741 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:19:12.743+0000 7f507149a7c0 -1 Falling back to public interface 2024-05-15T13:19:13.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:19:14.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:19:14.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:14.004 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:19:14.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:19:14.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:14.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:19:14.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:19:15.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:19:15.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:15.433 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:19:15.434 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:19:15.434 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:15.434 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:19:15.514 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:19:15.511+0000 7f507149a7c0 -1 osd.6 0 log_to_monitors true 2024-05-15T13:19:16.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:19:17.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:19:17.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:17.057 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:19:17.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:19:17.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:17.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:19:17.796 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:19:18.797 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:19:18.798 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:18.798 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:19:18.798 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:19:18.798 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:18.798 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:19:19.500 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:19:20.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:19:20.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:20.502 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:19:20.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:19:20.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:20.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:19:20.947 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:19:21.948 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:19:21.948 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:21.949 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:19:21.949 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:19:21.949 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:21.949 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:19:22.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:19:23.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:19:23.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:23.488 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:19:23.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:19:23.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:23.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:19:23.967 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:19:24.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:19:24.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:24.970 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:19:24.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:19:24.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:24.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:19:25.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:19:26.637 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:19:26.637 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:26.637 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:19:26.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:19:26.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:26.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:19:27.070 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:19:28.071 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:19:28.072 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:28.072 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:19:28.072 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:19:28.072 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:28.072 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:19:28.538 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:19:29.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:19:29.540 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:29.540 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:19:29.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:19:29.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:29.541 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:19:30.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:19:31.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:19:31.007 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:31.007 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:19:31.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:19:31.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:31.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:19:32.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:19:33.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:19:33.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:33.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:19:33.295 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:19:33.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:33.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:19:34.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:19:35.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:19:35.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:35.438 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:19:35.438 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:19:35.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:35.439 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:19:36.739 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:19:37.741 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:19:37.741 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:37.741 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:19:37.742 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:19:37.742 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:37.742 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:19:38.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:19:39.972 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:19:39.972 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:39.973 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:19:39.973 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:19:39.973 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:39.973 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:19:41.147 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:19:42.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:19:42.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:42.149 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:19:42.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:19:42.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:42.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:19:42.616 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:19:43.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:19:43.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:19:43.618 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:19:43.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:19:43.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:19:43.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-15T13:19:44.164 INFO:tasks.workunit.client.0.smithi005.stdout:osd.6 up in weight 1 up_from 59 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1855348736,v1:127.0.0.1:6851/1855348736] [v2:127.0.0.1:6852/1855348736,v1:127.0.0.1:6853/1855348736] exists,up 063e2e4e-c1a7-40f9-bcd6-f71d6d377737 2024-05-15T13:19:44.164 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:19:44.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:19:44.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:19:44.165 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-15T13:19:44.165 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T13:19:44.165 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T13:19:44.166 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:19:44.166 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:19:44.166 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:19:44.166 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.0.asok 2024-05-15T13:19:44.167 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-15T13:19:44.167 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-osd.0.asok log flush 2024-05-15T13:19:44.293 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T13:19:44.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-15T13:19:44.314 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T13:15:41.131+0000 7f23002227c0 0 load: jerasure load: lrc 2024-05-15T13:19:44.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:395: TEST_ec_single_recovery_error: local poolname=pool-jerasure 2024-05-15T13:19:44.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:396: TEST_ec_single_recovery_error: create_erasure_coded_pool pool-jerasure 3 2 2024-05-15T13:19:44.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-15T13:19:44.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-15T13:19:44.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-05-15T13:19:44.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-15T13:19:44.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-05-15T13:19:44.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-15T13:19:44.315 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-05-15T13:19:45.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-15T13:19:45.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-15T13:19:46.242 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-jerasure' already exists 2024-05-15T13:19:46.262 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T13:19:47.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-15T13:19:47.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:19:47.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:19:47.265 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:19:47.265 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:19:47.267 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:19:47.267 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:19:47.267 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:19:47.267 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:19:47.267 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:19:47.398 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:19:47.398 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:19:47.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:19:47.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:19:47.399 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:19:47.399 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:19:48.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:19:48.027 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:19:48.027 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:19:48.027 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T13:19:48.027 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T13:19:48.027 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T13:19:48.027 INFO:tasks.workunit.client.0.smithi005.stderr:6' 2024-05-15T13:19:48.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:19:48.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:19:48.027 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:19:48.191 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803826 2024-05-15T13:19:48.191 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803826 2024-05-15T13:19:48.191 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826' 2024-05-15T13:19:48.191 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:19:48.191 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:19:48.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=51539607599 2024-05-15T13:19:48.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 51539607599 2024-05-15T13:19:48.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-51539607599' 2024-05-15T13:19:48.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:19:48.344 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:19:48.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=77309411370 2024-05-15T13:19:48.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 77309411370 2024-05-15T13:19:48.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-51539607599 2-77309411370' 2024-05-15T13:19:48.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:19:48.489 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:19:48.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084320 2024-05-15T13:19:48.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084320 2024-05-15T13:19:48.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-51539607599 2-77309411370 3-120259084320' 2024-05-15T13:19:48.638 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:19:48.639 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T13:19:48.787 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757268 2024-05-15T13:19:48.787 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757268 2024-05-15T13:19:48.787 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-51539607599 2-77309411370 3-120259084320 4-163208757268' 2024-05-15T13:19:48.788 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:19:48.788 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-15T13:19:48.936 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397521 2024-05-15T13:19:48.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397521 2024-05-15T13:19:48.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-51539607599 2-77309411370 3-120259084320 4-163208757268 5-210453397521' 2024-05-15T13:19:48.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:19:48.937 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-15T13:19:49.093 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070472 2024-05-15T13:19:49.093 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070472 2024-05-15T13:19:49.093 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-51539607599 2-77309411370 3-120259084320 4-163208757268 5-210453397521 6-253403070472' 2024-05-15T13:19:49.093 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:19:49.094 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803826 2024-05-15T13:19:49.094 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:19:49.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:19:49.095 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803826 2024-05-15T13:19:49.095 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:19:49.097 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803826 2024-05-15T13:19:49.097 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 25769803826 2024-05-15T13:19:49.097 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803826' 2024-05-15T13:19:49.097 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:19:49.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803826 -lt 25769803826 2024-05-15T13:19:49.564 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:19:49.565 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-51539607599 2024-05-15T13:19:49.565 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:19:49.567 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:19:49.567 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-51539607599 2024-05-15T13:19:49.567 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:19:49.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=51539607599 2024-05-15T13:19:49.569 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 51539607599 2024-05-15T13:19:49.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 51539607599' 2024-05-15T13:19:49.569 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:19:50.023 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 51539607599 -lt 51539607599 2024-05-15T13:19:50.023 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:19:50.023 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-77309411370 2024-05-15T13:19:50.024 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:19:50.025 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:19:50.026 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-77309411370 2024-05-15T13:19:50.026 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:19:50.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=77309411370 2024-05-15T13:19:50.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 77309411370' 2024-05-15T13:19:50.028 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 77309411370 2024-05-15T13:19:50.028 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:19:50.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411370 -lt 77309411370 2024-05-15T13:19:50.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:19:50.478 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:19:50.478 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084320 2024-05-15T13:19:50.479 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:19:50.480 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084320 2024-05-15T13:19:50.480 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:19:50.481 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084320 2024-05-15T13:19:50.482 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 120259084320 2024-05-15T13:19:50.482 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084320' 2024-05-15T13:19:50.482 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:19:50.887 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084320 -lt 120259084320 2024-05-15T13:19:50.887 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:19:50.888 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-163208757268 2024-05-15T13:19:50.888 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:19:50.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T13:19:50.891 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-163208757268 2024-05-15T13:19:50.891 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:19:50.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757268 2024-05-15T13:19:50.892 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.4 seq 163208757268 2024-05-15T13:19:50.893 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 163208757268' 2024-05-15T13:19:50.893 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T13:19:51.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757268 -lt 163208757268 2024-05-15T13:19:51.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:19:51.336 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-210453397521 2024-05-15T13:19:51.336 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:19:51.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-15T13:19:51.338 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-210453397521 2024-05-15T13:19:51.339 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:19:51.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397521 2024-05-15T13:19:51.340 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.5 seq 210453397521 2024-05-15T13:19:51.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 210453397521' 2024-05-15T13:19:51.340 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-15T13:19:51.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397521 -lt 210453397521 2024-05-15T13:19:51.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:19:51.806 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:19:51.806 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-253403070472 2024-05-15T13:19:51.807 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-15T13:19:51.808 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-253403070472 2024-05-15T13:19:51.808 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:19:51.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070472 2024-05-15T13:19:51.809 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.6 seq 253403070472 2024-05-15T13:19:51.809 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 253403070472' 2024-05-15T13:19:51.810 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-15T13:19:52.321 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070472 -lt 253403070472 2024-05-15T13:19:52.322 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:19:52.322 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:19:52.322 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:19:52.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:19:52.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:19:52.878 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:19:52.878 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:19:52.878 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:19:52.878 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:19:52.878 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:19:52.878 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:19:53.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:19:53.288 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:19:53.289 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:19:53.289 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:19:53.889 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:19:53.889 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:19:53.889 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:19:53.889 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:398: TEST_ec_single_recovery_error: rados_put td/test-erasure-eio pool-jerasure myobject 2024-05-15T13:19:53.889 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-15T13:19:53.889 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-15T13:19:53.889 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-05-15T13:19:53.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:19:53.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-15T13:19:53.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:19:53.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-15T13:19:53.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:19:53.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-15T13:19:53.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:19:53.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-15T13:19:53.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-05-15T13:19:54.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:399: TEST_ec_single_recovery_error: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-05-15T13:19:54.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-15T13:19:54.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-15T13:19:54.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-15T13:19:54.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-15T13:19:54.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-15T13:19:54.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-15T13:19:54.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-05-15T13:19:54.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-15T13:19:54.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-15T13:19:54.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-15T13:19:54.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-05-15T13:19:54.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-15T13:19:54.116 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-05-15T13:19:54.116 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T13:19:54.116 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-05-15T13:19:54.116 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-05-15T13:19:54.116 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T13:19:54.554 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T13:19:54.554 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T13:19:54.554 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:19:54.554 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T13:19:54.554 INFO:tasks.workunit.client.0.smithi005.stderr:6' 2024-05-15T13:19:54.554 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-05-15T13:19:54.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2024-05-15T13:19:54.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-15T13:19:54.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-05-15T13:19:54.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-15T13:19:54.555 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-05-15T13:19:54.556 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-15T13:19:54.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-05-15T13:19:54.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-15T13:19:54.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-05-15T13:19:54.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-15T13:19:54.557 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-15T13:19:54.558 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-15T13:19:54.558 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-05-15T13:19:54.558 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-15T13:19:54.558 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-15T13:19:54.558 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:19:54.558 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:19:54.558 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:19:54.559 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.3.asok 2024-05-15T13:19:54.559 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.80702/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-05-15T13:19:54.688 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-15T13:19:54.688 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-15T13:19:54.689 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-15T13:19:54.689 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-05-15T13:19:54.689 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-15T13:19:54.690 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-15T13:19:54.690 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:19:54.690 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:19:54.691 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:19:54.691 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.3.asok 2024-05-15T13:19:54.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-15T13:19:54.691 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-05-15T13:19:54.812 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: get_osds pool-jerasure myobject 2024-05-15T13:19:54.812 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T13:19:54.812 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-05-15T13:19:54.813 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-05-15T13:19:54.813 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T13:19:55.285 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T13:19:55.285 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T13:19:55.285 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:19:55.285 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T13:19:55.285 INFO:tasks.workunit.client.0.smithi005.stderr:6' 2024-05-15T13:19:55.285 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-05-15T13:19:55.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: initial_osds=('3' '5' '1' '0' '6') 2024-05-15T13:19:55.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: local -a initial_osds 2024-05-15T13:19:55.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:402: TEST_ec_single_recovery_error: local last_osd=6 2024-05-15T13:19:55.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:404: TEST_ec_single_recovery_error: kill_daemons td/test-erasure-eio TERM osd.6 2024-05-15T13:19:55.286 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:19:55.286 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:19:55.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:19:55.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:19:55.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:19:55.596 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:19:55.596 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:405: TEST_ec_single_recovery_error: ceph osd down 6 2024-05-15T13:19:56.173 INFO:tasks.workunit.client.0.smithi005.stderr:osd.6 is already down. 2024-05-15T13:19:56.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:406: TEST_ec_single_recovery_error: ceph osd out 6 2024-05-15T13:19:56.814 INFO:tasks.workunit.client.0.smithi005.stderr:osd.6 is already out. 2024-05-15T13:19:56.832 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:409: TEST_ec_single_recovery_error: wait_for_clean 2024-05-15T13:19:56.832 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:19:56.832 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:19:56.832 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:19:56.833 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:19:56.834 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:19:56.834 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:19:56.834 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:19:56.834 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:19:56.834 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:19:56.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:19:56.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:19:56.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:19:56.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:19:56.950 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:19:56.950 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:19:57.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:19:57.685 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:19:57.685 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:19:57.685 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T13:19:57.685 INFO:tasks.workunit.client.0.smithi005.stderr:4 2024-05-15T13:19:57.685 INFO:tasks.workunit.client.0.smithi005.stderr:5 2024-05-15T13:19:57.685 INFO:tasks.workunit.client.0.smithi005.stderr:6' 2024-05-15T13:19:57.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:19:57.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:19:57.685 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:19:57.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803830 2024-05-15T13:19:57.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803830 2024-05-15T13:19:57.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830' 2024-05-15T13:19:57.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:19:57.917 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:19:58.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=51539607603 2024-05-15T13:19:58.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 51539607603 2024-05-15T13:19:58.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-51539607603' 2024-05-15T13:19:58.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:19:58.077 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:19:58.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=77309411374 2024-05-15T13:19:58.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 77309411374 2024-05-15T13:19:58.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-51539607603 2-77309411374' 2024-05-15T13:19:58.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:19:58.241 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:19:58.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084324 2024-05-15T13:19:58.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084324 2024-05-15T13:19:58.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-51539607603 2-77309411374 3-120259084324' 2024-05-15T13:19:58.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:19:58.403 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-15T13:19:58.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757272 2024-05-15T13:19:58.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757272 2024-05-15T13:19:58.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-51539607603 2-77309411374 3-120259084324 4-163208757272' 2024-05-15T13:19:58.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:19:58.569 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-15T13:19:58.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397525 2024-05-15T13:19:58.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397525 2024-05-15T13:19:58.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-51539607603 2-77309411374 3-120259084324 4-163208757272 5-210453397525' 2024-05-15T13:19:58.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:19:58.728 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-15T13:19:58.853 INFO:tasks.workunit.client.0.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-05-15T13:19:58.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-05-15T13:19:58.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-05-15T13:19:58.856 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-05-15T13:19:58.856 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:19:58.856 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803830 2024-05-15T13:19:58.856 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:19:58.857 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:19:58.858 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803830 2024-05-15T13:19:58.858 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:19:58.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803830 2024-05-15T13:19:58.859 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 25769803830 2024-05-15T13:19:58.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803830' 2024-05-15T13:19:58.859 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:19:59.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803830 -lt 25769803830 2024-05-15T13:19:59.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:19:59.329 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-51539607603 2024-05-15T13:19:59.329 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:19:59.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:19:59.331 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-51539607603 2024-05-15T13:19:59.332 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:19:59.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=51539607603 2024-05-15T13:19:59.333 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 51539607603 2024-05-15T13:19:59.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 51539607603' 2024-05-15T13:19:59.334 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:19:59.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 51539607603 -lt 51539607603 2024-05-15T13:19:59.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:19:59.776 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-77309411374 2024-05-15T13:19:59.776 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:19:59.777 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:19:59.779 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:19:59.779 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-77309411374 2024-05-15T13:19:59.780 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=77309411374 2024-05-15T13:19:59.780 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 77309411374' 2024-05-15T13:19:59.780 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 77309411374 2024-05-15T13:19:59.780 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:20:00.230 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411374 -lt 77309411374 2024-05-15T13:20:00.230 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:20:00.231 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084324 2024-05-15T13:20:00.232 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:20:00.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:20:00.234 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084324 2024-05-15T13:20:00.234 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:20:00.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084324 2024-05-15T13:20:00.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084324' 2024-05-15T13:20:00.235 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 120259084324 2024-05-15T13:20:00.236 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:20:00.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084324 -lt 120259084324 2024-05-15T13:20:00.700 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:20:00.701 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-163208757272 2024-05-15T13:20:00.701 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:20:00.702 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-15T13:20:00.703 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:20:00.704 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-163208757272 2024-05-15T13:20:00.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757272 2024-05-15T13:20:00.705 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.4 seq 163208757272 2024-05-15T13:20:00.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 163208757272' 2024-05-15T13:20:00.705 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-15T13:20:01.135 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757272 -lt 163208757272 2024-05-15T13:20:01.136 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:20:01.136 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-210453397525 2024-05-15T13:20:01.137 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:20:01.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-15T13:20:01.139 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-210453397525 2024-05-15T13:20:01.139 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:20:01.140 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397525 2024-05-15T13:20:01.140 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.5 seq 210453397525 2024-05-15T13:20:01.141 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 210453397525' 2024-05-15T13:20:01.141 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-15T13:20:01.580 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397525 -lt 210453397525 2024-05-15T13:20:01.589 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:20:01.589 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:20:01.589 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:20:02.139 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:20:02.139 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:20:02.139 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:20:02.140 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:20:02.140 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:20:02.140 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:20:02.140 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:20:02.140 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:20:02.566 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:20:02.566 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:20:02.567 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:20:02.567 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:20:03.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:20:03.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:20:03.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:20:03.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:411: TEST_ec_single_recovery_error: rados_get td/test-erasure-eio pool-jerasure myobject 2024-05-15T13:20:03.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-15T13:20:03.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-15T13:20:03.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-05-15T13:20:03.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-15T13:20:03.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-15T13:20:03.100 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-05-15T13:20:03.159 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-15T13:20:03.159 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-15T13:20:03.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:413: TEST_ec_single_recovery_error: delete_erasure_coded_pool pool-jerasure 2024-05-15T13:20:03.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-15T13:20:03.161 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-15T13:20:03.757 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-jerasure' does not exist 2024-05-15T13:20:03.777 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-15T13:20:04.702 INFO:tasks.workunit.client.0.smithi005.stderr:erasure-code-profile myprofile does not exist 2024-05-15T13:20:04.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-15T13:20:04.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-15T13:20:04.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T13:20:04.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-15T13:20:04.721 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:20:04.721 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:20:04.721 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:20:04.721 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:20:04.721 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:20:05.035 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:20:05.036 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T13:20:05.037 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T13:20:05.038 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T13:20:05.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T13:20:05.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T13:20:05.040 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T13:20:05.041 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:20:05.041 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T13:20:05.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T13:20:05.042 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:20:05.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:20:05.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T13:20:05.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-15T13:20:05.087 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T13:20:05.087 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:20:05.087 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:20:05.088 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80702 2024-05-15T13:20:05.089 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T13:20:05.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T13:20:05.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-15T13:20:05.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-15T13:20:05.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-15T13:20:05.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-15T13:20:05.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-15T13:20:05.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T13:20:05.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-15T13:20:05.091 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:20:05.091 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:20:05.091 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:20:05.091 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:20:05.091 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:20:05.093 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:20:05.094 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T13:20:05.094 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T13:20:05.095 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T13:20:05.096 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T13:20:05.096 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T13:20:05.097 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T13:20:05.097 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:20:05.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T13:20:05.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T13:20:05.098 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:20:05.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:20:05.101 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T13:20:05.101 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-15T13:20:05.102 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T13:20:05.102 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:20:05.102 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:20:05.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80702 2024-05-15T13:20:05.103 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T13:20:05.103 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T13:20:05.104 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-15T13:20:05.105 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T13:20:05.105 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:20:05.105 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:20:05.105 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.80702 2024-05-15T13:20:05.108 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T13:20:05.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-15T13:20:05.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T13:20:05.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-15T13:20:05.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-15T13:20:05.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-15T13:20:05.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T13:20:05.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T13:20:05.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T13:20:05.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-15T13:20:05.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-15T13:20:05.530 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T13:20:05.530 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:20:05.530 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:20:05.531 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:20:05.531 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:20:05.531 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:20:05.531 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:20:05.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-15T13:20:05.733 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T13:20:05.735 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T13:20:05.735 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T13:20:05.735 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T13:20:05.735 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T13:20:05.736 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T13:20:05.736 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T13:20:05.736 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T13:20:05.736 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T13:20:05.737 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:20:05.737 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:20:05.737 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:20:05.737 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T13:20:05.737 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T13:20:05.737 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.80702/ceph-mon.a.asok config get fsid 2024-05-15T13:20:05.855 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T13:20:05.855 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T13:20:05.855 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T13:20:05.855 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T13:20:05.856 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T13:20:05.856 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T13:20:05.856 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T13:20:05.856 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T13:20:05.857 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:20:05.857 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:20:05.857 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:20:05.857 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T13:20:05.857 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T13:20:05.857 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.80702/ceph-mon.a.asok config get mon_host 2024-05-15T13:20:05.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-15T13:20:05.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-15T13:20:05.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T13:20:05.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T13:20:05.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T13:20:05.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-15T13:20:05.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T13:20:06.881 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T13:20:06.881 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:20:06.881 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:20:06.881 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:20:06.881 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:20:06.881 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:20:06.882 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:20:06.882 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T13:20:06.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T13:20:06.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-15T13:20:06.913 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-15T13:20:07.211 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd' already exists 2024-05-15T13:20:07.227 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T13:20:08.230 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-15T13:20:08.230 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T13:20:08.230 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T13:20:08.231 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:20:08.231 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:20:08.231 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:20:08.231 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T13:20:08.232 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-15T13:20:08.232 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-mon.a.asok log flush 2024-05-15T13:20:08.332 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T13:20:08.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-15T13:20:08.347 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T13:20:05.578+0000 7f49ff464c80 0 load: jerasure load: lrc 2024-05-15T13:20:08.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_0 td/test-erasure-eio 2024-05-15T13:20:08.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:302: TEST_rados_get_bad_size_shard_0: local dir=td/test-erasure-eio 2024-05-15T13:20:08.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:303: TEST_rados_get_bad_size_shard_0: setup_osds 4 2024-05-15T13:20:08.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-05-15T13:20:08.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-15T13:20:08.349 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-05-15T13:20:08.350 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-05-15T13:20:08.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:20:08.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-15T13:20:08.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:20:08.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:20:08.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T13:20:08.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:20:08.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-15T13:20:08.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:20:08.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:20:08.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:20:08.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:20:08.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-15T13:20:08.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-15T13:20:08.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:20:08.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:20:08.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:20:08.356 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:20:08.356 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:20:08.356 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:20:08.356 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:20:08.356 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:20:08.356 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:20:08.356 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:20:08.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:20:08.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:20:08.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:20:08.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:20:08.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:20:08.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:20:08.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:20:08.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:20:08.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:20:08.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:20:08.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:20:08.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:20:08.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-15T13:20:08.357 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:20:08.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6ca7a839-0b13-408a-82b1-75624cc428f7 2024-05-15T13:20:08.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6ca7a839-0b13-408a-82b1-75624cc428f7' 2024-05-15T13:20:08.358 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 6ca7a839-0b13-408a-82b1-75624cc428f7 2024-05-15T13:20:08.358 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:20:08.373 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCItkRmhDdlFhAAD8cmubZvXwX0mcg4wl8mHQ== 2024-05-15T13:20:08.373 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCItkRmhDdlFhAAD8cmubZvXwX0mcg4wl8mHQ=="}' 2024-05-15T13:20:08.373 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6ca7a839-0b13-408a-82b1-75624cc428f7 -i td/test-erasure-eio/0/new.json 2024-05-15T13:20:08.713 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:20:08.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-15T13:20:08.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCItkRmhDdlFhAAD8cmubZvXwX0mcg4wl8mHQ== --osd-uuid 6ca7a839-0b13-408a-82b1-75624cc428f7 2024-05-15T13:20:08.765 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:08.766+0000 7fbb499be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:20:08.790 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:08.790+0000 7fbb499be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:20:08.791 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:08.790+0000 7fbb499be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:20:08.792 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:08.790+0000 7fbb499be7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T13:20:08.792 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:08.790+0000 7fbb499be7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T13:20:08.792 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:08.790+0000 7fbb499be7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T13:20:08.830 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:08.830+0000 7fbb499be7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-15T13:20:15.817 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-15T13:20:15.817 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:20:15.818 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T13:20:15.818 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T13:20:15.818 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:20:16.371 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T13:20:16.371 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T13:20:16.372 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:20:16.372 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:20:16.372 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:20:16.372 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:20:16.397 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:16.398+0000 7f1c7d3c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:20:16.400 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:16.402+0000 7f1c7d3c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:20:16.401 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:16.402+0000 7f1c7d3c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:20:16.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T13:20:16.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:20:16.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T13:20:16.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:20:16.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:20:16.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:16.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:20:16.776 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:20:16.776 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:16.776 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:20:16.957 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:16.958+0000 7f1c7d3c97c0 -1 Falling back to public interface 2024-05-15T13:20:17.202 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:18.204 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:18.204 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:18.204 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:20:18.205 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:20:18.205 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:18.205 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:20:18.620 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:19.621 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:19.621 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:19.622 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:20:19.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:20:19.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:20:19.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:19.622 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:19.622+0000 7f1c7d3c97c0 -1 osd.0 0 log_to_monitors true 2024-05-15T13:20:20.158 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:21.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:21.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:21.161 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:20:21.161 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:20:21.161 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:21.161 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:20:21.608 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:22.610 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:22.610 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:20:22.610 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:22.610 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:20:22.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:22.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:20:23.068 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:24.070 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:24.070 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:24.070 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:20:24.071 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:20:24.071 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:24.071 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:20:24.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:25.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:25.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:25.630 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:20:25.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:20:25.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:25.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:20:26.065 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:27.067 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:27.067 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:27.067 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:20:27.068 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:20:27.068 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:27.068 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:20:27.637 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:28.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:28.639 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:28.640 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:20:28.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:20:28.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:28.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:20:29.083 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:30.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:30.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:30.085 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:20:30.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:20:30.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:30.086 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:20:30.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:31.581 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:31.581 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:31.581 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:20:31.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:20:31.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:31.582 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:20:32.166 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:33.168 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:33.168 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:33.168 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:20:33.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:20:33.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:33.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:20:33.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:34.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:34.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:34.766 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:20:34.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:20:34.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:34.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:20:35.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:36.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:36.623 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:36.624 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:20:36.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:20:36.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:36.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:20:37.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:38.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:38.488 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:38.488 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:20:38.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:20:38.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:38.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:20:39.554 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:40.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:40.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:40.556 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:20:40.556 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:20:40.556 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:40.556 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:20:41.657 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:42.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:42.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:42.659 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:20:42.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:20:42.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:42.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:20:43.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:44.621 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:44.621 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:44.622 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:20:44.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:20:44.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:44.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:20:45.174 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:46.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:46.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:46.177 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:20:46.177 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:20:46.177 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:46.177 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:20:46.669 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/685678473,v1:127.0.0.1:6803/685678473] [v2:127.0.0.1:6804/685678473,v1:127.0.0.1:6805/685678473] exists,up 6ca7a839-0b13-408a-82b1-75624cc428f7 2024-05-15T13:20:46.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:20:46.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:20:46.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:20:46.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:20:46.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-15T13:20:46.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:20:46.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:20:46.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-15T13:20:46.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:20:46.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-15T13:20:46.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:20:46.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:20:46.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:20:46.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:20:46.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-15T13:20:46.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-15T13:20:46.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:20:46.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:20:46.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:20:46.673 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:20:46.673 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:20:46.673 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:20:46.673 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:20:46.673 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:20:46.673 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:20:46.674 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:20:46.674 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:20:46.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:20:46.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:20:46.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:20:46.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:20:46.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:20:46.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:20:46.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:20:46.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:20:46.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:20:46.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:20:46.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:20:46.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-15T13:20:46.675 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:20:46.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=13fc1548-9ac0-4594-9e6f-fb2a2b817cbd 2024-05-15T13:20:46.676 INFO:tasks.workunit.client.0.smithi005.stdout:add osd1 13fc1548-9ac0-4594-9e6f-fb2a2b817cbd 2024-05-15T13:20:46.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 13fc1548-9ac0-4594-9e6f-fb2a2b817cbd' 2024-05-15T13:20:46.676 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:20:46.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCutkRmymTvKRAA+QhPsnVo1zzH4y6bRWrgSA== 2024-05-15T13:20:46.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCutkRmymTvKRAA+QhPsnVo1zzH4y6bRWrgSA=="}' 2024-05-15T13:20:46.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 13fc1548-9ac0-4594-9e6f-fb2a2b817cbd -i td/test-erasure-eio/1/new.json 2024-05-15T13:20:47.205 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:20:47.222 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-15T13:20:47.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCutkRmymTvKRAA+QhPsnVo1zzH4y6bRWrgSA== --osd-uuid 13fc1548-9ac0-4594-9e6f-fb2a2b817cbd 2024-05-15T13:20:47.253 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:47.254+0000 7f0c901fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:20:47.291 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:47.290+0000 7f0c901fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:20:47.293 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:47.294+0000 7f0c901fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:20:47.293 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:47.294+0000 7f0c901fd7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T13:20:47.293 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:47.294+0000 7f0c901fd7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T13:20:47.294 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:47.294+0000 7f0c901fd7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T13:20:47.331 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:47.330+0000 7f0c901fd7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-15T13:20:54.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-15T13:20:54.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:20:54.005 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd1 key to auth repository 2024-05-15T13:20:54.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-15T13:20:54.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:20:54.570 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.1 2024-05-15T13:20:54.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-15T13:20:54.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:20:54.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:20:54.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:20:54.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:20:54.603 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:54.602+0000 7fef800257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:20:54.647 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:54.646+0000 7fef800257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:20:54.649 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:54.650+0000 7fef800257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:20:54.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-15T13:20:54.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:20:54.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-15T13:20:54.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:20:54.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:20:54.969 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:54.970 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:20:54.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:20:54.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:20:54.970 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:55.222 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:55.222+0000 7fef800257c0 -1 Falling back to public interface 2024-05-15T13:20:55.431 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:56.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:56.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:56.433 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:20:56.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:20:56.433 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:56.434 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:20:56.853 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:57.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:57.855 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:57.856 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:20:57.856 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:20:57.856 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:57.856 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:20:57.884 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:20:57.885+0000 7fef800257c0 -1 osd.1 0 log_to_monitors true 2024-05-15T13:20:58.321 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:20:59.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:20:59.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:20:59.323 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:20:59.324 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:20:59.324 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:20:59.324 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:20:59.870 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:00.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:00.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:00.872 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:21:00.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:21:00.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:00.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:21:01.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:02.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:02.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:02.311 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:21:02.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:21:02.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:02.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:21:02.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:03.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:03.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:03.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:21:03.873 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:21:03.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:03.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:21:04.354 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:05.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:05.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:05.356 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:21:05.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:21:05.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:05.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:21:05.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:06.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:06.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:06.874 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:21:06.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:21:06.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:06.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:21:07.304 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:08.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:08.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:08.306 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:21:08.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:21:08.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:08.307 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:21:08.873 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:09.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:09.875 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:09.876 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:21:09.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:21:09.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:09.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:21:10.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:11.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:11.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:11.463 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:21:11.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:21:11.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:11.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:21:12.131 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:13.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:13.132 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:13.133 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:21:13.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:21:13.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:13.133 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:21:13.577 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:14.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:14.579 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:14.579 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:21:14.580 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:21:14.580 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:14.580 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:21:15.932 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:16.935 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:16.935 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:16.935 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:21:16.936 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:21:16.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:16.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:21:17.792 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:18.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:18.794 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:18.795 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:21:18.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:21:18.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:18.795 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:21:19.341 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:20.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:20.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:20.343 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:21:20.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:21:20.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:20.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:21:21.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:22.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:22.485 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:22.485 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:21:22.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:21:22.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:22.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:21:23.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:24.379 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:24.379 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:24.379 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:21:24.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:21:24.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:24.380 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:21:24.886 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:25.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:25.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:25.889 INFO:tasks.workunit.client.0.smithi005.stdout:19 2024-05-15T13:21:25.889 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-15T13:21:25.889 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:25.889 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:21:26.341 INFO:tasks.workunit.client.0.smithi005.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/342490914,v1:127.0.0.1:6811/342490914] [v2:127.0.0.1:6812/342490914,v1:127.0.0.1:6813/342490914] exists,up 13fc1548-9ac0-4594-9e6f-fb2a2b817cbd 2024-05-15T13:21:26.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:21:26.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:21:26.343 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:21:26.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:21:26.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-15T13:21:26.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:21:26.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:21:26.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-15T13:21:26.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:21:26.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-15T13:21:26.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:21:26.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:21:26.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:21:26.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:21:26.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-15T13:21:26.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-15T13:21:26.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:21:26.344 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:21:26.345 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:21:26.345 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:21:26.345 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:21:26.345 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:21:26.345 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:21:26.345 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:21:26.345 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:21:26.346 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:21:26.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:21:26.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:21:26.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:21:26.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:21:26.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:21:26.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:21:26.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:21:26.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:21:26.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:21:26.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:21:26.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:21:26.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:21:26.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-15T13:21:26.347 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:21:26.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=defc26b5-22b4-4f91-bfbc-12d06f0e245d 2024-05-15T13:21:26.348 INFO:tasks.workunit.client.0.smithi005.stdout:add osd2 defc26b5-22b4-4f91-bfbc-12d06f0e245d 2024-05-15T13:21:26.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 defc26b5-22b4-4f91-bfbc-12d06f0e245d' 2024-05-15T13:21:26.348 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:21:26.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDWtkRmNr7rFRAAtgtbxLoi/uX0J/cvJ+TrHw== 2024-05-15T13:21:26.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDWtkRmNr7rFRAAtgtbxLoi/uX0J/cvJ+TrHw=="}' 2024-05-15T13:21:26.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new defc26b5-22b4-4f91-bfbc-12d06f0e245d -i td/test-erasure-eio/2/new.json 2024-05-15T13:21:27.002 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:21:27.018 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-15T13:21:27.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDWtkRmNr7rFRAAtgtbxLoi/uX0J/cvJ+TrHw== --osd-uuid defc26b5-22b4-4f91-bfbc-12d06f0e245d 2024-05-15T13:21:27.054 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:21:27.053+0000 7f8862f247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:21:27.056 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:21:27.057+0000 7f8862f247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:21:27.058 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:21:27.057+0000 7f8862f247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:21:27.058 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:21:27.057+0000 7f8862f247c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T13:21:27.058 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:21:27.057+0000 7f8862f247c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T13:21:27.058 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:21:27.057+0000 7f8862f247c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T13:21:27.145 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:21:27.145+0000 7f8862f247c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-15T13:21:34.093 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-15T13:21:34.093 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:21:34.094 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd2 key to auth repository 2024-05-15T13:21:34.094 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-15T13:21:34.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:21:34.652 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.2 2024-05-15T13:21:34.653 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-15T13:21:34.653 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:21:34.653 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:21:34.653 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:21:34.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:21:34.678 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:21:34.677+0000 7f91de4477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:21:34.680 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:21:34.681+0000 7f91de4477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:21:34.682 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:21:34.681+0000 7f91de4477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:21:35.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-15T13:21:35.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:21:35.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-15T13:21:35.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:21:35.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:21:35.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:35.160 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:21:35.161 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:21:35.161 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:35.161 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:21:35.248 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:21:35.245+0000 7f91de4477c0 -1 Falling back to public interface 2024-05-15T13:21:35.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:36.615 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:36.615 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:36.616 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:21:36.616 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:21:36.616 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:36.616 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:21:37.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:37.928 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:21:37.925+0000 7f91de4477c0 -1 osd.2 0 log_to_monitors true 2024-05-15T13:21:38.046 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:38.046 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:38.047 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:21:38.047 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:21:38.047 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:38.047 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:21:38.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:39.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:39.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:39.708 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:21:39.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:21:39.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:39.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:21:40.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:41.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:41.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:41.330 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:21:41.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:21:41.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:41.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:21:41.868 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:42.870 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:42.870 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:42.870 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:21:42.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:21:42.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:42.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:21:43.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:44.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:44.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:44.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:21:44.335 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:21:44.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:44.336 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:21:44.834 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:45.836 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:45.836 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:45.836 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:21:45.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:21:45.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:45.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:21:46.347 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:47.349 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:47.349 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:47.349 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:21:47.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:21:47.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:47.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:21:47.835 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:48.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:48.837 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:48.837 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:21:48.838 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:21:48.838 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:48.838 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:21:49.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:50.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:50.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:50.364 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:21:50.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:21:50.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:50.365 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:21:50.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:51.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:51.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:51.933 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:21:51.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:21:51.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:51.934 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:21:52.867 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:53.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:53.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:53.869 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:21:53.870 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:21:53.870 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:53.870 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:21:54.422 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:55.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:55.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:55.426 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:21:55.426 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:21:55.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:55.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:21:56.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:57.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:57.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:57.261 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:21:57.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:21:57.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:57.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:21:58.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:21:59.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:21:59.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:21:59.045 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:21:59.045 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:21:59.046 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:21:59.046 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:21:59.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:00.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:00.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:00.884 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:22:00.885 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:22:00.885 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:00.885 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:22:01.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:02.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:02.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:02.831 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:22:02.831 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:22:02.831 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:02.831 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:22:03.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:04.647 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:04.647 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:04.647 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:22:04.648 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:22:04.648 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:04.648 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:22:05.223 INFO:tasks.workunit.client.0.smithi005.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/823007550,v1:127.0.0.1:6819/823007550] [v2:127.0.0.1:6820/823007550,v1:127.0.0.1:6821/823007550] exists,up defc26b5-22b4-4f91-bfbc-12d06f0e245d 2024-05-15T13:22:05.227 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:22:05.227 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:22:05.227 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:22:05.227 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:22:05.227 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-15T13:22:05.227 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:22:05.227 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:22:05.227 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-15T13:22:05.227 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:22:05.227 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-15T13:22:05.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:22:05.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:22:05.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:22:05.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:22:05.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-15T13:22:05.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-15T13:22:05.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:22:05.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:22:05.228 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:22:05.228 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:22:05.228 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:22:05.228 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:22:05.228 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:22:05.228 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:22:05.228 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:22:05.228 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:22:05.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:22:05.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:22:05.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:22:05.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:22:05.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:22:05.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:22:05.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:22:05.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:22:05.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:22:05.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:22:05.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:22:05.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:22:05.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-15T13:22:05.230 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:22:05.230 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=88fdbdeb-1be5-4922-9a2d-6149a04a85f6 2024-05-15T13:22:05.230 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 88fdbdeb-1be5-4922-9a2d-6149a04a85f6' 2024-05-15T13:22:05.230 INFO:tasks.workunit.client.0.smithi005.stdout:add osd3 88fdbdeb-1be5-4922-9a2d-6149a04a85f6 2024-05-15T13:22:05.231 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:22:05.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD9tkRmRvshDxAAFIUAk16gruvuH7bNFKHuhQ== 2024-05-15T13:22:05.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD9tkRmRvshDxAAFIUAk16gruvuH7bNFKHuhQ=="}' 2024-05-15T13:22:05.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 88fdbdeb-1be5-4922-9a2d-6149a04a85f6 -i td/test-erasure-eio/3/new.json 2024-05-15T13:22:05.889 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:22:05.905 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-15T13:22:05.906 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD9tkRmRvshDxAAFIUAk16gruvuH7bNFKHuhQ== --osd-uuid 88fdbdeb-1be5-4922-9a2d-6149a04a85f6 2024-05-15T13:22:05.941 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:22:05.940+0000 7fddcb8f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:22:05.952 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:22:05.948+0000 7fddcb8f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:22:05.953 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:22:05.952+0000 7fddcb8f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:22:05.954 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:22:05.952+0000 7fddcb8f67c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T13:22:05.954 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:22:05.952+0000 7fddcb8f67c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T13:22:05.954 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:22:05.952+0000 7fddcb8f67c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T13:22:06.161 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:22:06.156+0000 7fddcb8f67c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-15T13:22:12.682 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-15T13:22:12.682 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:22:12.683 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd3 key to auth repository 2024-05-15T13:22:12.683 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-15T13:22:12.683 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:22:13.265 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.3 2024-05-15T13:22:13.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-15T13:22:13.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:22:13.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:22:13.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:22:13.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:22:13.299 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:22:13.296+0000 7f00bff397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:22:13.326 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:22:13.324+0000 7f00bff397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:22:13.327 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:22:13.324+0000 7f00bff397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:22:13.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-15T13:22:13.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:22:13.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-15T13:22:13.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:22:13.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:22:13.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:13.701 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:22:13.701 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:22:13.702 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:13.702 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:13.898 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:22:13.896+0000 7f00bff397c0 -1 Falling back to public interface 2024-05-15T13:22:14.168 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:15.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:15.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:15.170 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:22:15.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:22:15.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:15.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:15.600 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:16.598 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:22:16.596+0000 7f00bff397c0 -1 osd.3 0 log_to_monitors true 2024-05-15T13:22:16.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:16.601 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:16.601 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:22:16.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:22:16.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:16.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:17.192 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:18.194 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:18.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:18.195 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:22:18.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:22:18.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:18.196 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:18.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:19.939 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:19.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:19.940 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:22:19.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:22:19.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:19.941 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:20.528 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:21.530 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:21.530 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:21.530 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:22:21.530 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:22:21.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:21.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:22.020 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:23.022 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:23.022 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:23.022 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:22:23.023 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:22:23.023 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:23.023 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:23.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:24.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:24.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:24.549 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:22:24.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:22:24.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:24.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:25.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:26.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:26.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:26.004 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:22:26.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:22:26.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:26.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:26.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:27.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:27.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:27.548 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:22:27.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:22:27.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:27.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:28.052 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:29.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:29.054 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:29.054 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:22:29.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:22:29.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:29.055 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:29.530 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:30.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:30.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:30.532 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:22:30.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:22:30.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:30.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:31.082 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:32.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:32.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:32.084 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:22:32.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:22:32.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:32.085 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:32.935 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:33.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:33.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:33.938 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:22:33.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:22:33.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:33.938 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:34.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:35.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:35.924 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:35.925 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:22:35.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:22:35.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:35.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:36.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:37.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:37.427 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:37.428 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:22:37.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:22:37.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:37.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:38.189 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:39.190 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:39.190 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:39.191 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:22:39.191 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:22:39.191 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:39.191 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:40.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:41.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:41.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:41.006 INFO:tasks.workunit.client.0.smithi005.stdout:17 2024-05-15T13:22:41.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-15T13:22:41.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:41.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:41.637 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:42.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:42.640 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:42.640 INFO:tasks.workunit.client.0.smithi005.stdout:18 2024-05-15T13:22:42.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-15T13:22:42.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:42.641 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:43.111 INFO:tasks.workunit.client.0.smithi005.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4040024676,v1:127.0.0.1:6827/4040024676] [v2:127.0.0.1:6828/4040024676,v1:127.0.0.1:6829/4040024676] exists,up 88fdbdeb-1be5-4922-9a2d-6149a04a85f6 2024-05-15T13:22:43.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:22:43.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:22:43.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:22:43.113 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-15T13:22:43.113 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T13:22:43.113 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T13:22:43.114 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:22:43.114 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:22:43.114 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:22:43.114 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.0.asok 2024-05-15T13:22:43.114 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-15T13:22:43.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-osd.0.asok log flush 2024-05-15T13:22:43.222 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T13:22:43.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-15T13:22:43.240 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T13:20:16.962+0000 7f1c7d3c97c0 0 load: jerasure load: lrc 2024-05-15T13:22:43.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:305: TEST_rados_get_bad_size_shard_0: local poolname=pool-jerasure 2024-05-15T13:22:43.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:306: TEST_rados_get_bad_size_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-05-15T13:22:43.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-15T13:22:43.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-15T13:22:43.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-05-15T13:22:43.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-15T13:22:43.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-05-15T13:22:43.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-15T13:22:43.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-05-15T13:22:43.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-15T13:22:43.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-15T13:22:45.601 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-jerasure' already exists 2024-05-15T13:22:45.622 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T13:22:46.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-15T13:22:46.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:22:46.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:22:46.624 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:22:46.625 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:22:46.626 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:22:46.626 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:22:46.627 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:22:46.627 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:22:46.627 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:22:46.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:22:46.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:22:46.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:22:46.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:22:46.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:22:46.762 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:22:47.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:22:47.200 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:22:47.200 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:22:47.201 INFO:tasks.workunit.client.0.smithi005.stderr:3' 2024-05-15T13:22:47.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:22:47.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:22:47.201 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:22:47.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803810 2024-05-15T13:22:47.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803810 2024-05-15T13:22:47.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803810' 2024-05-15T13:22:47.440 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:22:47.441 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:22:47.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=51539607574 2024-05-15T13:22:47.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 51539607574 2024-05-15T13:22:47.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803810 1-51539607574' 2024-05-15T13:22:47.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:22:47.599 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:22:47.781 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378641 2024-05-15T13:22:47.782 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378641 2024-05-15T13:22:47.782 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803810 1-51539607574 2-81604378641' 2024-05-15T13:22:47.782 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:22:47.782 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:22:47.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084294 2024-05-15T13:22:47.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084294 2024-05-15T13:22:47.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803810 1-51539607574 2-81604378641 3-120259084294' 2024-05-15T13:22:47.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:22:47.931 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803810 2024-05-15T13:22:47.931 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:22:47.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:22:47.932 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803810 2024-05-15T13:22:47.932 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:22:47.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803810 2024-05-15T13:22:47.933 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 25769803810 2024-05-15T13:22:47.933 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803810' 2024-05-15T13:22:47.933 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:22:48.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803810 -lt 25769803810 2024-05-15T13:22:48.350 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:22:48.351 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-51539607574 2024-05-15T13:22:48.351 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:22:48.353 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:22:48.354 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-51539607574 2024-05-15T13:22:48.354 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:22:48.355 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=51539607574 2024-05-15T13:22:48.356 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 51539607574' 2024-05-15T13:22:48.356 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 51539607574 2024-05-15T13:22:48.356 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:22:48.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 51539607574 -lt 51539607574 2024-05-15T13:22:48.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:22:48.879 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378641 2024-05-15T13:22:48.879 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:22:48.881 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:22:48.882 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378641 2024-05-15T13:22:48.882 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:22:48.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378641 2024-05-15T13:22:48.883 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 81604378641 2024-05-15T13:22:48.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378641' 2024-05-15T13:22:48.884 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:22:49.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378639 -lt 81604378641 2024-05-15T13:22:49.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-15T13:22:50.314 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-15T13:22:50.315 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:22:50.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378642 -lt 81604378641 2024-05-15T13:22:50.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:22:50.747 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084294 2024-05-15T13:22:50.747 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:22:50.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:22:50.749 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084294 2024-05-15T13:22:50.749 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:22:50.753 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084294 2024-05-15T13:22:50.754 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084294' 2024-05-15T13:22:50.754 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 120259084294 2024-05-15T13:22:50.754 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:22:51.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084295 -lt 120259084294 2024-05-15T13:22:51.181 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:22:51.182 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:22:51.182 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:22:51.774 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:22:51.774 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:22:51.775 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:22:51.775 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:22:51.775 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:22:51.775 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:22:51.775 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:22:51.775 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:22:52.239 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:22:52.239 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:22:52.240 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:22:52.240 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:22:52.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:22:52.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:22:52.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:22:52.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:308: TEST_rados_get_bad_size_shard_0: local shard_id=0 2024-05-15T13:22:52.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:309: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 10 2024-05-15T13:22:52.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-05-15T13:22:52.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-05-15T13:22:52.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-05-15T13:22:52.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-05-15T13:22:52.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=10 2024-05-15T13:22:52.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-05-15T13:22:52.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-05-15T13:22:52.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-05-15T13:22:52.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-80702-0-10 2024-05-15T13:22:52.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-80702-0-10 2024-05-15T13:22:52.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-15T13:22:52.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-15T13:22:52.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-80702-0-10 2024-05-15T13:22:52.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:22:52.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-15T13:22:52.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:22:52.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-15T13:22:52.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:22:52.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-15T13:22:52.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:22:52.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-15T13:22:52.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-80702-0-10 td/test-erasure-eio/ORIGINAL 2024-05-15T13:22:52.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-80702-0-10 td/test-erasure-eio 0 10 set 2024-05-15T13:22:52.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-80702-0-10 2024-05-15T13:22:52.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-05-15T13:22:52.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-05-15T13:22:52.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-05-15T13:22:52.944 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-05-15T13:22:52.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-05-15T13:22:52.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-05-15T13:22:52.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-05-15T13:22:52.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-05-15T13:22:52.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-05-15T13:22:52.945 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-80702-0-10 2024-05-15T13:22:52.945 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T13:22:52.945 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-80702-0-10 2024-05-15T13:22:52.945 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-80702-0-10 2024-05-15T13:22:52.946 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T13:22:53.389 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T13:22:53.389 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:22:53.390 INFO:tasks.workunit.client.0.smithi005.stderr:2' 2024-05-15T13:22:53.390 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-15T13:22:53.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-05-15T13:22:53.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-05-15T13:22:53.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-05-15T13:22:53.390 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-05-15T13:22:54.093 INFO:tasks.workunit.client.0.smithi005.stderr:noout is set 2024-05-15T13:22:54.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-05-15T13:22:54.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-05-15T13:22:54.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-05-15T13:22:54.115 INFO:tasks.workunit.client.0.smithi005.stderr:1+0 records in 2024-05-15T13:22:54.115 INFO:tasks.workunit.client.0.smithi005.stderr:1+0 records out 2024-05-15T13:22:54.115 INFO:tasks.workunit.client.0.smithi005.stderr:10 bytes copied, 0.000120746 s, 82.8 kB/s 2024-05-15T13:22:54.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-80702-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:22:54.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-15T13:22:54.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-15T13:22:54.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-05-15T13:22:54.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-15T13:22:54.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-80702-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:22:54.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-15T13:22:54.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-15T13:22:54.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-05-15T13:22:54.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-15T13:22:54.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-05-15T13:22:54.116 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:22:54.116 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:22:54.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:22:54.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:22:54.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:22:55.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:22:55.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-80702-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:22:55.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-15T13:22:55.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-15T13:22:55.428 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-05-15T13:22:55.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-15T13:22:55.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-05-15T13:22:55.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-80702-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:22:57.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-05-15T13:22:57.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-15T13:22:57.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-15T13:22:57.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-05-15T13:22:57.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-15T13:22:57.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-05-15T13:22:57.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:22:57.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:22:57.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:22:57.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:22:57.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-15T13:22:57.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-15T13:22:57.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-15T13:22:57.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-15T13:22:57.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:22:57.462 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-15T13:22:57.463 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:22:57.463 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:22:57.463 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:22:57.463 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:22:57.463 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:22:57.463 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:22:57.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:22:57.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:22:57.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:22:57.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:22:57.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:22:57.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:22:57.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:22:57.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:22:57.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-15T13:22:57.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-15T13:22:57.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-05-15T13:22:57.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-05-15T13:22:57.466 INFO:tasks.workunit.client.0.smithi005.stderr:start osd.3 2024-05-15T13:22:57.466 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:22:57.466 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-05-15T13:22:57.466 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-05-15T13:22:57.467 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-15T13:22:57.467 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-15T13:22:57.468 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-15T13:22:57.501 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:22:57.499+0000 7f88739c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:22:57.504 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:22:57.503+0000 7f88739c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:22:57.505 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:22:57.507+0000 7f88739c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:22:57.902 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-05-15T13:22:57.902 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:22:57.902 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-15T13:22:57.902 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:22:57.903 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:22:57.903 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:57.903 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:22:57.903 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T13:22:57.903 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:57.903 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:58.058 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:22:58.059+0000 7f88739c27c0 -1 Falling back to public interface 2024-05-15T13:22:58.303 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:22:59.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:22:59.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:22:59.305 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:22:59.305 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:22:59.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:22:59.306 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:22:59.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:23:00.665 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:23:00.667+0000 7f88739c27c0 -1 osd.3 35 log_to_monitors true 2024-05-15T13:23:00.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:23:00.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:23:00.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:23:00.720 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:23:00.721 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:23:00.721 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:23:01.269 INFO:tasks.workunit.client.0.smithi005.stderr:osd.3 up in weight 1 up_from 40 up_thru 32 down_at 37 last_clean_interval [28,35) [v2:127.0.0.1:6826/85429669,v1:127.0.0.1:6827/85429669] [v2:127.0.0.1:6828/85429669,v1:127.0.0.1:6829/85429669] exists,up 88fdbdeb-1be5-4922-9a2d-6149a04a85f6 2024-05-15T13:23:01.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:23:01.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:23:01.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:23:01.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-15T13:23:01.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:23:01.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:23:01.269 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:23:01.270 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:23:01.270 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:23:01.270 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:23:01.271 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:23:01.271 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:23:01.271 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:23:01.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:23:01.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:23:01.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:23:01.391 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:23:01.392 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:23:01.392 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:23:01.973 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:23:01.973 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:23:01.973 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:23:01.973 INFO:tasks.workunit.client.0.smithi005.stderr:3' 2024-05-15T13:23:01.973 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:23:01.974 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:23:01.974 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:23:02.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803815 2024-05-15T13:23:02.202 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803815 2024-05-15T13:23:02.202 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803815' 2024-05-15T13:23:02.202 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:23:02.202 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:23:02.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=51539607579 2024-05-15T13:23:02.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 51539607579 2024-05-15T13:23:02.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803815 1-51539607579' 2024-05-15T13:23:02.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:23:02.536 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:23:02.702 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378646 2024-05-15T13:23:02.702 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378646 2024-05-15T13:23:02.702 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803815 1-51539607579 2-81604378646' 2024-05-15T13:23:02.702 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:23:02.702 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:23:02.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691843 2024-05-15T13:23:02.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691843 2024-05-15T13:23:02.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803815 1-51539607579 2-81604378646 3-171798691843' 2024-05-15T13:23:02.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:23:02.925 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803815 2024-05-15T13:23:02.926 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:23:02.927 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:23:02.928 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803815 2024-05-15T13:23:02.928 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:23:02.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803815 2024-05-15T13:23:02.930 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803815' 2024-05-15T13:23:02.930 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.0 seq 25769803815 2024-05-15T13:23:02.930 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:23:03.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803813 -lt 25769803815 2024-05-15T13:23:03.366 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-15T13:23:04.368 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-15T13:23:04.369 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:23:04.801 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803815 -lt 25769803815 2024-05-15T13:23:04.802 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:23:04.802 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-51539607579 2024-05-15T13:23:04.803 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:23:04.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:23:04.806 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:23:04.806 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-51539607579 2024-05-15T13:23:04.808 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=51539607579 2024-05-15T13:23:04.808 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 51539607579' 2024-05-15T13:23:04.808 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.1 seq 51539607579 2024-05-15T13:23:04.808 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:23:05.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 51539607579 -lt 51539607579 2024-05-15T13:23:05.233 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:23:05.234 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:23:05.234 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378646 2024-05-15T13:23:05.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:23:05.237 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378646 2024-05-15T13:23:05.237 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:23:05.238 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378646 2024-05-15T13:23:05.239 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378646' 2024-05-15T13:23:05.239 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.2 seq 81604378646 2024-05-15T13:23:05.239 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:23:05.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378646 -lt 81604378646 2024-05-15T13:23:05.661 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:23:05.662 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-171798691843 2024-05-15T13:23:05.662 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:23:05.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:23:05.665 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-171798691843 2024-05-15T13:23:05.665 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:23:05.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691843 2024-05-15T13:23:05.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 171798691843' 2024-05-15T13:23:05.666 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.3 seq 171798691843 2024-05-15T13:23:05.666 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:23:06.118 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691843 -lt 171798691843 2024-05-15T13:23:06.119 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:23:06.120 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:23:06.120 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:23:06.719 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:23:06.719 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:23:06.720 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:23:06.720 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:23:06.720 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:23:06.720 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:23:06.720 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:23:06.720 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:23:07.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-15T13:23:07.171 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:23:07.171 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:23:07.171 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:23:07.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-05-15T13:23:07.713 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-05-15T13:23:07.713 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-05-15T13:23:07.713 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-05-15T13:23:07.713 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-15T13:23:07.713 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-05-15T13:23:07.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-15T13:23:07.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:23:07.814 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:23:07.815 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:23:07.815 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:23:07.815 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:23:07.815 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:23:07.815 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:23:08.229 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:23:08.229 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:23:08.230 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:23:08.230 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:23:08.771 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:23:08.771 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:23:08.771 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:23:08.772 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-05-15T13:23:08.772 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-05-15T13:23:09.513 INFO:tasks.workunit.client.0.smithi005.stderr:noout is unset 2024-05-15T13:23:09.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-80702-0-10 2024-05-15T13:23:09.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-15T13:23:09.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-15T13:23:09.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-80702-0-10 2024-05-15T13:23:09.535 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-15T13:23:09.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-15T13:23:09.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-80702-0-10 td/test-erasure-eio/COPY 2024-05-15T13:23:09.650 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-15T13:23:09.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-15T13:23:09.653 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-05-15T13:23:09.657 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-05-15T13:23:09.657 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-80702-0-10 td/test-erasure-eio 1 10 set 2024-05-15T13:23:09.657 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-80702-0-10 2024-05-15T13:23:09.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-05-15T13:23:09.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-05-15T13:23:09.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-05-15T13:23:09.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-05-15T13:23:09.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-05-15T13:23:09.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-05-15T13:23:09.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-05-15T13:23:09.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-05-15T13:23:09.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-05-15T13:23:09.659 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-80702-0-10 2024-05-15T13:23:09.659 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T13:23:09.659 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-80702-0-10 2024-05-15T13:23:09.660 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T13:23:09.660 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-80702-0-10 2024-05-15T13:23:10.162 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T13:23:10.162 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:23:10.162 INFO:tasks.workunit.client.0.smithi005.stderr:2' 2024-05-15T13:23:10.162 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-15T13:23:10.162 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-05-15T13:23:10.162 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-05-15T13:23:10.162 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-05-15T13:23:10.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-05-15T13:23:10.827 INFO:tasks.workunit.client.0.smithi005.stderr:noout is set 2024-05-15T13:23:10.846 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-05-15T13:23:10.846 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-05-15T13:23:10.847 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-05-15T13:23:10.849 INFO:tasks.workunit.client.0.smithi005.stderr:1+0 records in 2024-05-15T13:23:10.849 INFO:tasks.workunit.client.0.smithi005.stderr:1+0 records out 2024-05-15T13:23:10.849 INFO:tasks.workunit.client.0.smithi005.stderr:10 bytes copied, 0.0001944 s, 51.4 kB/s 2024-05-15T13:23:10.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-80702-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:23:10.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-15T13:23:10.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-15T13:23:10.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-05-15T13:23:10.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-15T13:23:10.849 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-80702-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:23:10.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-15T13:23:10.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-15T13:23:10.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-05-15T13:23:10.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-15T13:23:10.850 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-05-15T13:23:10.851 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:23:10.851 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:23:10.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:23:10.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:23:10.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:23:12.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:23:12.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-80702-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:23:12.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-15T13:23:12.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-15T13:23:12.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-05-15T13:23:12.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-15T13:23:12.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-05-15T13:23:12.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-80702-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:23:14.725 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-05-15T13:23:14.725 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-15T13:23:14.725 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-15T13:23:14.725 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-05-15T13:23:14.725 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-15T13:23:14.725 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-05-15T13:23:14.725 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:23:14.726 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:23:14.726 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:23:14.726 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:23:14.726 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-15T13:23:14.726 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-15T13:23:14.726 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-15T13:23:14.726 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-15T13:23:14.726 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:23:14.726 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-15T13:23:14.726 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:23:14.726 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:23:14.728 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:23:14.728 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:23:14.728 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:23:14.728 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:23:14.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:23:14.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:23:14.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:23:14.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:23:14.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:23:14.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:23:14.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:23:14.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:23:14.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-15T13:23:14.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-15T13:23:14.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-05-15T13:23:14.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-05-15T13:23:14.729 INFO:tasks.workunit.client.0.smithi005.stderr:start osd.1 2024-05-15T13:23:14.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:23:14.730 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-05-15T13:23:14.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-05-15T13:23:14.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-15T13:23:14.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-15T13:23:14.731 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-15T13:23:14.764 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:23:14.763+0000 7ff45208f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:23:14.767 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:23:14.767+0000 7ff45208f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:23:14.769 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:23:14.767+0000 7ff45208f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:23:15.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-05-15T13:23:15.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:23:15.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-15T13:23:15.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:23:15.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:23:15.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:23:15.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:23:15.169 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T13:23:15.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:23:15.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:23:15.311 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:23:15.311+0000 7ff45208f7c0 -1 Falling back to public interface 2024-05-15T13:23:15.610 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:23:16.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:23:16.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:23:16.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:23:16.613 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:23:16.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:23:16.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:23:17.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:23:18.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:23:18.078 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:23:18.078 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:23:18.078 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:23:18.078 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:23:18.079 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:23:18.229 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:23:18.231+0000 7ff45208f7c0 -1 osd.1 46 log_to_monitors true 2024-05-15T13:23:18.583 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:23:19.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:23:19.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:23:19.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:23:19.586 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T13:23:19.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:23:19.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:23:20.331 INFO:tasks.workunit.client.0.smithi005.stderr:osd.1 up in weight 1 up_from 51 up_thru 51 down_at 47 last_clean_interval [12,46) [v2:127.0.0.1:6810/2253080644,v1:127.0.0.1:6811/2253080644] [v2:127.0.0.1:6812/2253080644,v1:127.0.0.1:6813/2253080644] exists,up 13fc1548-9ac0-4594-9e6f-fb2a2b817cbd 2024-05-15T13:23:20.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:23:20.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:23:20.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:23:20.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-15T13:23:20.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:23:20.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:23:20.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:23:20.332 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:23:20.332 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:23:20.333 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:23:20.333 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:23:20.333 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:23:20.333 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:23:20.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:23:20.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:23:20.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:23:20.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:23:20.449 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:23:20.449 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:23:20.941 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:23:20.941 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:23:20.941 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:23:20.941 INFO:tasks.workunit.client.0.smithi005.stderr:3' 2024-05-15T13:23:20.941 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:23:20.941 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:23:20.942 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:23:21.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803820 2024-05-15T13:23:21.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803820 2024-05-15T13:23:21.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803820' 2024-05-15T13:23:21.099 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:23:21.099 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:23:21.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=219043332099 2024-05-15T13:23:21.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 219043332099 2024-05-15T13:23:21.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803820 1-219043332099' 2024-05-15T13:23:21.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:23:21.293 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:23:21.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378652 2024-05-15T13:23:21.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378652 2024-05-15T13:23:21.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803820 1-219043332099 2-81604378652' 2024-05-15T13:23:21.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:23:21.447 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:23:21.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691849 2024-05-15T13:23:21.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691849 2024-05-15T13:23:21.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803820 1-219043332099 2-81604378652 3-171798691849' 2024-05-15T13:23:21.602 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:23:21.602 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803820 2024-05-15T13:23:21.602 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:23:21.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:23:21.605 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803820 2024-05-15T13:23:21.605 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:23:21.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803820 2024-05-15T13:23:21.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803820' 2024-05-15T13:23:21.606 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.0 seq 25769803820 2024-05-15T13:23:21.606 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:23:22.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803821 -lt 25769803820 2024-05-15T13:23:22.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:23:22.028 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-219043332099 2024-05-15T13:23:22.028 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:23:22.030 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:23:22.031 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-219043332099 2024-05-15T13:23:22.031 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:23:22.032 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=219043332099 2024-05-15T13:23:22.032 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 219043332099' 2024-05-15T13:23:22.032 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.1 seq 219043332099 2024-05-15T13:23:22.033 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:23:22.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 219043332099 -lt 219043332099 2024-05-15T13:23:22.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:23:22.461 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378652 2024-05-15T13:23:22.462 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:23:22.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:23:22.465 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378652 2024-05-15T13:23:22.465 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:23:22.466 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378652 2024-05-15T13:23:22.466 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378652' 2024-05-15T13:23:22.466 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.2 seq 81604378652 2024-05-15T13:23:22.467 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:23:22.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378652 -lt 81604378652 2024-05-15T13:23:22.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:23:22.884 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:23:22.884 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-171798691849 2024-05-15T13:23:22.885 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:23:22.886 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-171798691849 2024-05-15T13:23:22.886 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:23:22.887 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691849 2024-05-15T13:23:22.887 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 171798691849' 2024-05-15T13:23:22.888 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.3 seq 171798691849 2024-05-15T13:23:22.888 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:23:23.312 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691849 -lt 171798691849 2024-05-15T13:23:23.312 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:23:23.313 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:23:23.313 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:23:23.896 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:23:23.896 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:23:23.897 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:23:23.897 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:23:23.897 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:23:23.897 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:23:23.897 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:23:23.897 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:23:24.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:23:24.314 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:23:24.314 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:23:24.315 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:23:24.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:23:24.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:23:24.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:23:24.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-05-15T13:23:24.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-05-15T13:23:25.658 INFO:tasks.workunit.client.0.smithi005.stderr:noout is unset 2024-05-15T13:23:25.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-80702-0-10 fail 2024-05-15T13:23:25.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-15T13:23:25.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-15T13:23:25.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-80702-0-10 2024-05-15T13:23:25.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-05-15T13:23:25.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-05-15T13:23:25.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-80702-0-10 td/test-erasure-eio/COPY 2024-05-15T13:23:25.873 INFO:tasks.workunit.client.0.smithi005.stderr:error getting pool-jerasure/obj-size-80702-0-10: (5) Input/output error 2024-05-15T13:23:25.878 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-05-15T13:23:25.878 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-05-15T13:23:25.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:310: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 0 2024-05-15T13:23:25.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-05-15T13:23:25.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-05-15T13:23:25.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-05-15T13:23:25.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-05-15T13:23:25.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=0 2024-05-15T13:23:25.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-05-15T13:23:25.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-05-15T13:23:25.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-05-15T13:23:25.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-80702-0-0 2024-05-15T13:23:25.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-80702-0-0 2024-05-15T13:23:25.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-15T13:23:25.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-15T13:23:25.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-80702-0-0 2024-05-15T13:23:25.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:23:25.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-15T13:23:25.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:23:25.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-15T13:23:25.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:23:25.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-15T13:23:25.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:23:25.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-15T13:23:25.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-80702-0-0 td/test-erasure-eio/ORIGINAL 2024-05-15T13:23:26.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-80702-0-0 td/test-erasure-eio 0 0 set 2024-05-15T13:23:26.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-80702-0-0 2024-05-15T13:23:26.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-05-15T13:23:26.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-05-15T13:23:26.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-05-15T13:23:26.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-05-15T13:23:26.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-05-15T13:23:26.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-05-15T13:23:26.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-05-15T13:23:26.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-05-15T13:23:26.165 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-05-15T13:23:26.166 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-80702-0-0 2024-05-15T13:23:26.166 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T13:23:26.166 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-80702-0-0 2024-05-15T13:23:26.166 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-80702-0-0 2024-05-15T13:23:26.166 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T13:23:26.645 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T13:23:26.645 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:23:26.645 INFO:tasks.workunit.client.0.smithi005.stderr:2' 2024-05-15T13:23:26.645 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-15T13:23:26.646 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-05-15T13:23:26.646 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-05-15T13:23:26.646 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-05-15T13:23:26.646 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-05-15T13:23:27.364 INFO:tasks.workunit.client.0.smithi005.stderr:noout is set 2024-05-15T13:23:27.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-05-15T13:23:27.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-05-15T13:23:27.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-05-15T13:23:27.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-80702-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:23:27.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-15T13:23:27.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-15T13:23:27.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-05-15T13:23:27.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-15T13:23:27.393 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-80702-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:23:27.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-15T13:23:27.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-15T13:23:27.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-05-15T13:23:27.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-15T13:23:27.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-05-15T13:23:27.394 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:23:27.394 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:23:27.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:23:27.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:23:27.394 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:23:28.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:23:28.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-80702-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:23:28.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-15T13:23:28.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-15T13:23:28.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-05-15T13:23:28.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-15T13:23:28.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-05-15T13:23:28.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-80702-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:23:30.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-05-15T13:23:30.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-15T13:23:30.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-15T13:23:30.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-05-15T13:23:30.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-15T13:23:30.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-05-15T13:23:30.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:23:30.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:23:30.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:23:30.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:23:30.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-15T13:23:30.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-15T13:23:30.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-15T13:23:30.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-15T13:23:30.628 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:23:30.628 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-15T13:23:30.628 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:23:30.628 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:23:30.629 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:23:30.629 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:23:30.629 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:23:30.629 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:23:30.629 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:23:30.629 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:23:30.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:23:30.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:23:30.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:23:30.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:23:30.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:23:30.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:23:30.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-15T13:23:30.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-15T13:23:30.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-05-15T13:23:30.630 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-05-15T13:23:30.631 INFO:tasks.workunit.client.0.smithi005.stderr:start osd.3 2024-05-15T13:23:30.631 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:23:30.631 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-05-15T13:23:30.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-05-15T13:23:30.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-15T13:23:30.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-15T13:23:30.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-15T13:23:30.667 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:23:30.667+0000 7f067b2077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:23:30.670 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:23:30.671+0000 7f067b2077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:23:30.671 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:23:30.671+0000 7f067b2077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:23:31.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-05-15T13:23:31.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:23:31.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-15T13:23:31.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:23:31.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:23:31.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:23:31.062 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:23:31.062 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T13:23:31.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:23:31.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:23:31.243 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:23:31.243+0000 7f067b2077c0 -1 Falling back to public interface 2024-05-15T13:23:31.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:23:32.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:23:32.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:23:32.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:23:32.545 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:23:32.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:23:32.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:23:32.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:23:33.977 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:23:33.977 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:23:33.977 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:23:33.977 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:23:33.978 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:23:33.978 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:23:33.980 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:23:33.979+0000 7f067b2077c0 -1 osd.3 58 log_to_monitors true 2024-05-15T13:23:34.581 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:23:35.583 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:23:35.583 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:23:35.583 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:23:35.583 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T13:23:35.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:23:35.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:23:36.276 INFO:tasks.workunit.client.0.smithi005.stderr:osd.3 up in weight 1 up_from 62 up_thru 62 down_at 59 last_clean_interval [40,58) [v2:127.0.0.1:6826/989969593,v1:127.0.0.1:6827/989969593] [v2:127.0.0.1:6828/989969593,v1:127.0.0.1:6829/989969593] exists,up 88fdbdeb-1be5-4922-9a2d-6149a04a85f6 2024-05-15T13:23:36.276 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:23:36.276 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:23:36.276 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:23:36.276 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-15T13:23:36.277 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:23:36.277 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:23:36.277 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:23:36.277 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:23:36.277 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:23:36.278 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:23:36.278 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:23:36.278 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:23:36.278 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:23:36.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:23:36.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:23:36.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:23:36.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:23:36.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:23:36.405 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:23:36.989 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:23:36.989 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:23:36.989 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:23:36.989 INFO:tasks.workunit.client.0.smithi005.stderr:3' 2024-05-15T13:23:36.989 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:23:36.989 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:23:36.990 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:23:37.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803826 2024-05-15T13:23:37.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803826 2024-05-15T13:23:37.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826' 2024-05-15T13:23:37.184 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:23:37.184 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:23:37.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=219043332104 2024-05-15T13:23:37.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 219043332104 2024-05-15T13:23:37.348 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-219043332104' 2024-05-15T13:23:37.349 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:23:37.349 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:23:37.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378657 2024-05-15T13:23:37.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378657 2024-05-15T13:23:37.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-219043332104 2-81604378657' 2024-05-15T13:23:37.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:23:37.505 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:23:37.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972355 2024-05-15T13:23:37.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972355 2024-05-15T13:23:37.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-219043332104 2-81604378657 3-266287972355' 2024-05-15T13:23:37.658 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:23:37.658 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803826 2024-05-15T13:23:37.659 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:23:37.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:23:37.661 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803826 2024-05-15T13:23:37.661 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:23:37.662 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803826 2024-05-15T13:23:37.663 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803826' 2024-05-15T13:23:37.663 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.0 seq 25769803826 2024-05-15T13:23:37.663 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:23:38.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803826 -lt 25769803826 2024-05-15T13:23:38.102 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:23:38.103 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-219043332104 2024-05-15T13:23:38.103 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:23:38.105 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:23:38.106 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:23:38.106 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-219043332104 2024-05-15T13:23:38.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=219043332104 2024-05-15T13:23:38.107 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 219043332104' 2024-05-15T13:23:38.108 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.1 seq 219043332104 2024-05-15T13:23:38.108 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:23:38.530 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 219043332104 -lt 219043332104 2024-05-15T13:23:38.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:23:38.531 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378657 2024-05-15T13:23:38.531 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:23:38.533 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:23:38.534 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378657 2024-05-15T13:23:38.534 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:23:38.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378657 2024-05-15T13:23:38.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378657' 2024-05-15T13:23:38.537 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.2 seq 81604378657 2024-05-15T13:23:38.537 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:23:38.955 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378657 -lt 81604378657 2024-05-15T13:23:38.955 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:23:38.956 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972355 2024-05-15T13:23:38.956 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:23:38.958 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:23:38.959 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972355 2024-05-15T13:23:38.959 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:23:38.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972355 2024-05-15T13:23:38.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972355' 2024-05-15T13:23:38.960 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.3 seq 266287972355 2024-05-15T13:23:38.961 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:23:39.404 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972355 -lt 266287972355 2024-05-15T13:23:39.404 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:23:39.405 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:23:39.405 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:23:39.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:23:39.962 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:23:39.962 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:23:39.962 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:23:39.962 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:23:39.962 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:23:39.963 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:23:39.963 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:23:40.389 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:23:40.390 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:23:40.390 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:23:40.391 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:23:40.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:23:40.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:23:40.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:23:40.931 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-05-15T13:23:40.932 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-05-15T13:23:41.613 INFO:tasks.workunit.client.0.smithi005.stderr:noout is unset 2024-05-15T13:23:41.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-80702-0-0 2024-05-15T13:23:41.635 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-15T13:23:41.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-15T13:23:41.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-80702-0-0 2024-05-15T13:23:41.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-15T13:23:41.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-15T13:23:41.636 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-80702-0-0 td/test-erasure-eio/COPY 2024-05-15T13:23:41.831 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-15T13:23:41.833 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-15T13:23:41.848 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-05-15T13:23:41.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-05-15T13:23:41.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-80702-0-0 td/test-erasure-eio 1 0 set 2024-05-15T13:23:41.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-80702-0-0 2024-05-15T13:23:41.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-05-15T13:23:41.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-05-15T13:23:41.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-05-15T13:23:41.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-05-15T13:23:41.851 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-05-15T13:23:41.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-05-15T13:23:41.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-05-15T13:23:41.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-05-15T13:23:41.852 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-05-15T13:23:41.852 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-80702-0-0 2024-05-15T13:23:41.852 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T13:23:41.852 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-80702-0-0 2024-05-15T13:23:41.853 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-80702-0-0 2024-05-15T13:23:41.853 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T13:23:42.287 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T13:23:42.287 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:23:42.287 INFO:tasks.workunit.client.0.smithi005.stderr:2' 2024-05-15T13:23:42.287 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-15T13:23:42.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-05-15T13:23:42.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-05-15T13:23:42.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-05-15T13:23:42.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-05-15T13:23:43.085 INFO:tasks.workunit.client.0.smithi005.stderr:noout is set 2024-05-15T13:23:43.108 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-05-15T13:23:43.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-05-15T13:23:43.109 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-05-15T13:23:43.111 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-80702-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:23:43.111 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-15T13:23:43.111 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-15T13:23:43.111 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-05-15T13:23:43.111 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-15T13:23:43.111 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-80702-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:23:43.111 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-15T13:23:43.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-15T13:23:43.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-05-15T13:23:43.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-15T13:23:43.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-05-15T13:23:43.112 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:23:43.112 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:23:43.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:23:43.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:23:43.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:23:44.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:23:44.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-80702-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:23:44.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-15T13:23:44.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-15T13:23:44.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-05-15T13:23:44.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-15T13:23:44.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-05-15T13:23:44.425 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-80702-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:23:46.253 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-05-15T13:23:46.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-15T13:23:46.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-15T13:23:46.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-05-15T13:23:46.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-15T13:23:46.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-05-15T13:23:46.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:23:46.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:23:46.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:23:46.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:23:46.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-15T13:23:46.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-15T13:23:46.254 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-15T13:23:46.255 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-15T13:23:46.255 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:23:46.255 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-15T13:23:46.255 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:23:46.255 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:23:46.255 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:23:46.255 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:23:46.255 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:23:46.256 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:23:46.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:23:46.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:23:46.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:23:46.256 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:23:46.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:23:46.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:23:46.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:23:46.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:23:46.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-15T13:23:46.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-15T13:23:46.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-05-15T13:23:46.257 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-05-15T13:23:46.258 INFO:tasks.workunit.client.0.smithi005.stderr:start osd.1 2024-05-15T13:23:46.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:23:46.258 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-05-15T13:23:46.258 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-05-15T13:23:46.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-15T13:23:46.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-15T13:23:46.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-15T13:23:46.292 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:23:46.291+0000 7f7a124aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:23:46.295 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:23:46.295+0000 7f7a124aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:23:46.296 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:23:46.295+0000 7f7a124aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:23:46.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-05-15T13:23:46.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:23:46.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-15T13:23:46.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:23:46.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:23:46.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:23:46.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:23:46.685 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T13:23:46.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:23:46.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:23:46.867 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:23:46.866+0000 7f7a124aa7c0 -1 Falling back to public interface 2024-05-15T13:23:47.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:23:48.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:23:48.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:23:48.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:23:48.112 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:23:48.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:23:48.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:23:48.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:23:49.482 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:23:49.482+0000 7f7a124aa7c0 -1 osd.1 67 log_to_monitors true 2024-05-15T13:23:49.572 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:23:49.572 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:23:49.572 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:23:49.572 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:23:49.573 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:23:49.573 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:23:50.070 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:23:51.072 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:23:51.072 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:23:51.072 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:23:51.072 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T13:23:51.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:23:51.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:23:51.676 INFO:tasks.workunit.client.0.smithi005.stderr:osd.1 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [51,67) [v2:127.0.0.1:6810/115678308,v1:127.0.0.1:6811/115678308] [v2:127.0.0.1:6812/115678308,v1:127.0.0.1:6813/115678308] exists,up 13fc1548-9ac0-4594-9e6f-fb2a2b817cbd 2024-05-15T13:23:51.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:23:51.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:23:51.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:23:51.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-15T13:23:51.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:23:51.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:23:51.676 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:23:51.676 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:23:51.677 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:23:51.677 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:23:51.678 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:23:51.678 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:23:51.678 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:23:51.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:23:51.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:23:51.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:23:51.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:23:51.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:23:51.805 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:23:52.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:23:52.436 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:23:52.436 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:23:52.436 INFO:tasks.workunit.client.0.smithi005.stderr:3' 2024-05-15T13:23:52.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:23:52.436 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:23:52.436 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:23:52.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803831 2024-05-15T13:23:52.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803831 2024-05-15T13:23:52.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803831' 2024-05-15T13:23:52.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:23:52.599 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:23:52.752 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645315 2024-05-15T13:23:52.752 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645315 2024-05-15T13:23:52.752 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803831 1-309237645315' 2024-05-15T13:23:52.753 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:23:52.753 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:23:52.907 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378662 2024-05-15T13:23:52.907 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378662 2024-05-15T13:23:52.907 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803831 1-309237645315 2-81604378662' 2024-05-15T13:23:52.907 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:23:52.907 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:23:53.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972360 2024-05-15T13:23:53.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972360 2024-05-15T13:23:53.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803831 1-309237645315 2-81604378662 3-266287972360' 2024-05-15T13:23:53.063 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:23:53.064 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803831 2024-05-15T13:23:53.064 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:23:53.065 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:23:53.065 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803831 2024-05-15T13:23:53.066 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:23:53.067 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803831 2024-05-15T13:23:53.067 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803831' 2024-05-15T13:23:53.067 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.0 seq 25769803831 2024-05-15T13:23:53.067 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:23:53.478 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803829 -lt 25769803831 2024-05-15T13:23:53.478 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-15T13:23:54.480 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-15T13:23:54.481 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:23:54.905 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803831 -lt 25769803831 2024-05-15T13:23:54.906 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:23:54.906 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-309237645315 2024-05-15T13:23:54.906 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:23:54.908 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:23:54.909 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-309237645315 2024-05-15T13:23:54.909 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:23:54.910 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645315 2024-05-15T13:23:54.910 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 309237645315' 2024-05-15T13:23:54.910 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.1 seq 309237645315 2024-05-15T13:23:54.911 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:23:55.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645315 -lt 309237645315 2024-05-15T13:23:55.330 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:23:55.331 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378662 2024-05-15T13:23:55.331 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:23:55.333 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:23:55.334 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:23:55.334 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378662 2024-05-15T13:23:55.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378662 2024-05-15T13:23:55.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378662' 2024-05-15T13:23:55.335 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.2 seq 81604378662 2024-05-15T13:23:55.336 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:23:55.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378662 -lt 81604378662 2024-05-15T13:23:55.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:23:55.763 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972360 2024-05-15T13:23:55.763 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:23:55.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:23:55.766 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972360 2024-05-15T13:23:55.766 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:23:55.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972360 2024-05-15T13:23:55.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972360' 2024-05-15T13:23:55.768 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.3 seq 266287972360 2024-05-15T13:23:55.768 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:23:56.316 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972361 -lt 266287972360 2024-05-15T13:23:56.317 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:23:56.318 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:23:56.318 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:23:56.867 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:23:56.867 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:23:56.868 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:23:56.868 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:23:56.868 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:23:56.868 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:23:56.868 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:23:56.868 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:23:57.311 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:23:57.312 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:23:57.312 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:23:57.312 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:23:57.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:23:57.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:23:57.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:23:57.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-05-15T13:23:57.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-05-15T13:23:58.632 INFO:tasks.workunit.client.0.smithi005.stderr:noout is unset 2024-05-15T13:23:58.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-80702-0-0 fail 2024-05-15T13:23:58.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-15T13:23:58.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-15T13:23:58.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-80702-0-0 2024-05-15T13:23:58.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-05-15T13:23:58.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-05-15T13:23:58.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-80702-0-0 td/test-erasure-eio/COPY 2024-05-15T13:23:58.850 INFO:tasks.workunit.client.0.smithi005.stderr:error getting pool-jerasure/obj-size-80702-0-0: (5) Input/output error 2024-05-15T13:23:58.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-05-15T13:23:58.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-05-15T13:23:58.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:311: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 256 add 2024-05-15T13:23:58.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-05-15T13:23:58.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-05-15T13:23:58.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-05-15T13:23:58.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-05-15T13:23:58.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=256 2024-05-15T13:23:58.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-05-15T13:23:58.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=add 2024-05-15T13:23:58.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-05-15T13:23:58.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-80702-0-256 2024-05-15T13:23:58.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-80702-0-256 2024-05-15T13:23:58.858 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-15T13:23:58.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-15T13:23:58.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-80702-0-256 2024-05-15T13:23:58.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:23:58.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-15T13:23:58.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:23:58.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-15T13:23:58.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:23:58.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-15T13:23:58.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:23:58.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-15T13:23:58.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-80702-0-256 td/test-erasure-eio/ORIGINAL 2024-05-15T13:23:59.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-80702-0-256 td/test-erasure-eio 0 256 add 2024-05-15T13:23:59.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-80702-0-256 2024-05-15T13:23:59.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-05-15T13:23:59.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-05-15T13:23:59.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-05-15T13:23:59.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-05-15T13:23:59.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-05-15T13:23:59.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-05-15T13:23:59.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-05-15T13:23:59.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-05-15T13:23:59.117 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-05-15T13:23:59.117 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-80702-0-256 2024-05-15T13:23:59.117 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T13:23:59.117 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-80702-0-256 2024-05-15T13:23:59.117 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-80702-0-256 2024-05-15T13:23:59.117 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T13:23:59.550 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T13:23:59.550 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:23:59.550 INFO:tasks.workunit.client.0.smithi005.stderr:2' 2024-05-15T13:23:59.550 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-15T13:23:59.551 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-05-15T13:23:59.551 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-05-15T13:23:59.551 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-05-15T13:23:59.551 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-05-15T13:24:00.222 INFO:tasks.workunit.client.0.smithi005.stderr:noout is set 2024-05-15T13:24:00.241 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-05-15T13:24:00.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-80702-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:24:00.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-15T13:24:00.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-15T13:24:00.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-05-15T13:24:00.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-15T13:24:00.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-80702-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:24:00.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-15T13:24:00.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-15T13:24:00.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-05-15T13:24:00.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-15T13:24:00.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-05-15T13:24:00.242 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:24:00.243 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:24:00.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:24:00.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:24:00.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:24:01.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:24:01.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-80702-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:24:01.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-15T13:24:01.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-15T13:24:01.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-05-15T13:24:01.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-15T13:24:01.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-05-15T13:24:01.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-80702-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:24:03.137 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-05-15T13:24:03.137 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-15T13:24:03.137 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-15T13:24:03.137 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-05-15T13:24:03.137 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-15T13:24:03.137 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-05-15T13:24:03.137 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:24:03.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:24:03.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:24:03.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:24:03.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-15T13:24:03.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-15T13:24:03.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-15T13:24:03.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-15T13:24:03.138 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:24:03.138 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-15T13:24:03.138 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:24:03.138 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:24:03.139 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:24:03.139 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:24:03.140 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:24:03.140 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:24:03.140 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:24:03.140 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:24:03.140 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:24:03.140 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:24:03.140 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:24:03.140 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:24:03.140 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:24:03.140 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:24:03.140 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-15T13:24:03.140 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-15T13:24:03.140 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-05-15T13:24:03.141 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-05-15T13:24:03.141 INFO:tasks.workunit.client.0.smithi005.stderr:start osd.3 2024-05-15T13:24:03.141 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:24:03.141 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-05-15T13:24:03.143 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-05-15T13:24:03.143 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-15T13:24:03.143 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-15T13:24:03.144 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-15T13:24:03.176 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:24:03.174+0000 7fd2294de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:24:03.179 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:24:03.178+0000 7fd2294de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:24:03.180 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:24:03.178+0000 7fd2294de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:24:03.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-05-15T13:24:03.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:24:03.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-15T13:24:03.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:24:03.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:24:03.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:24:03.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:24:03.633 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T13:24:03.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:24:03.634 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:24:03.767 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:24:03.766+0000 7fd2294de7c0 -1 Falling back to public interface 2024-05-15T13:24:04.057 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:24:05.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:24:05.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:24:05.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:24:05.059 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:24:05.059 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:24:05.060 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:24:05.489 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:24:06.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:24:06.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:24:06.491 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:24:06.491 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:24:06.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:24:06.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:24:06.553 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:24:06.550+0000 7fd2294de7c0 -1 osd.3 77 log_to_monitors true 2024-05-15T13:24:07.058 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:24:08.060 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:24:08.060 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:24:08.060 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:24:08.060 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T13:24:08.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:24:08.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:24:08.649 INFO:tasks.workunit.client.0.smithi005.stderr:osd.3 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [62,77) [v2:127.0.0.1:6826/3026860236,v1:127.0.0.1:6827/3026860236] [v2:127.0.0.1:6828/3026860236,v1:127.0.0.1:6829/3026860236] exists,up 88fdbdeb-1be5-4922-9a2d-6149a04a85f6 2024-05-15T13:24:08.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:24:08.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:24:08.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:24:08.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-15T13:24:08.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:24:08.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:24:08.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:24:08.650 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:24:08.650 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:24:08.650 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:24:08.651 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:24:08.651 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:24:08.651 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:24:08.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:24:08.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:24:08.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:24:08.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:24:08.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:24:08.768 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:24:09.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:24:09.199 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:24:09.199 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:24:09.199 INFO:tasks.workunit.client.0.smithi005.stderr:3' 2024-05-15T13:24:09.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:24:09.199 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:24:09.199 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:24:09.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803836 2024-05-15T13:24:09.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803836 2024-05-15T13:24:09.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803836' 2024-05-15T13:24:09.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:24:09.403 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:24:09.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645320 2024-05-15T13:24:09.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645320 2024-05-15T13:24:09.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803836 1-309237645320' 2024-05-15T13:24:09.568 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:24:09.568 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:24:09.722 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378668 2024-05-15T13:24:09.723 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378668 2024-05-15T13:24:09.723 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803836 1-309237645320 2-81604378668' 2024-05-15T13:24:09.723 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:24:09.723 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:24:09.887 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318275 2024-05-15T13:24:09.887 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318275 2024-05-15T13:24:09.887 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803836 1-309237645320 2-81604378668 3-352187318275' 2024-05-15T13:24:09.887 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:24:09.888 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803836 2024-05-15T13:24:09.888 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:24:09.888 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:24:09.889 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803836 2024-05-15T13:24:09.889 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:24:09.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803836 2024-05-15T13:24:09.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803836' 2024-05-15T13:24:09.891 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.0 seq 25769803836 2024-05-15T13:24:09.891 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:24:10.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803836 -lt 25769803836 2024-05-15T13:24:10.308 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:24:10.309 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-309237645320 2024-05-15T13:24:10.309 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:24:10.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:24:10.311 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-309237645320 2024-05-15T13:24:10.311 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:24:10.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645320 2024-05-15T13:24:10.313 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 309237645320' 2024-05-15T13:24:10.313 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.1 seq 309237645320 2024-05-15T13:24:10.313 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:24:10.743 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645320 -lt 309237645320 2024-05-15T13:24:10.743 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:24:10.744 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378668 2024-05-15T13:24:10.744 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:24:10.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:24:10.747 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378668 2024-05-15T13:24:10.747 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:24:10.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378668 2024-05-15T13:24:10.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378668' 2024-05-15T13:24:10.749 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.2 seq 81604378668 2024-05-15T13:24:10.749 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:24:11.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378668 -lt 81604378668 2024-05-15T13:24:11.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:24:11.171 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-352187318275 2024-05-15T13:24:11.172 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:24:11.174 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:24:11.175 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-352187318275 2024-05-15T13:24:11.175 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:24:11.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318275 2024-05-15T13:24:11.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 352187318275' 2024-05-15T13:24:11.176 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.3 seq 352187318275 2024-05-15T13:24:11.177 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:24:11.609 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 352187318275 2024-05-15T13:24:11.609 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-15T13:24:12.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-15T13:24:12.612 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:24:13.047 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318275 -lt 352187318275 2024-05-15T13:24:13.048 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:24:13.048 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:24:13.049 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:24:13.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:24:13.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:24:13.713 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:24:13.713 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:24:13.713 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:24:13.713 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:24:13.714 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:24:13.714 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:24:14.210 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:24:14.210 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:24:14.210 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:24:14.210 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:24:14.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:24:14.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:24:14.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:24:14.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-05-15T13:24:14.762 INFO:tasks.workunit.client.0.smithi005.stderr:1+0 records in 2024-05-15T13:24:14.762 INFO:tasks.workunit.client.0.smithi005.stderr:1+0 records out 2024-05-15T13:24:14.762 INFO:tasks.workunit.client.0.smithi005.stderr:256 bytes copied, 4.5249e-05 s, 5.7 MB/s 2024-05-15T13:24:14.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-80702-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:24:14.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-15T13:24:14.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-15T13:24:14.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-05-15T13:24:14.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-15T13:24:14.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-80702-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:24:14.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-15T13:24:14.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-15T13:24:14.762 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-05-15T13:24:14.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-15T13:24:14.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-05-15T13:24:14.763 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:24:14.763 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:24:14.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:24:14.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:24:14.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:24:15.072 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:24:15.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-80702-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:24:15.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-15T13:24:15.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-15T13:24:15.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-05-15T13:24:15.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-15T13:24:15.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-05-15T13:24:15.073 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-80702-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:24:17.316 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-05-15T13:24:17.316 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-15T13:24:17.316 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-15T13:24:17.316 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-05-15T13:24:17.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-15T13:24:17.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-05-15T13:24:17.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:24:17.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:24:17.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:24:17.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:24:17.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-15T13:24:17.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-15T13:24:17.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-15T13:24:17.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-15T13:24:17.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:24:17.319 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-15T13:24:17.319 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:24:17.319 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:24:17.319 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:24:17.319 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:24:17.319 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:24:17.319 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:24:17.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:24:17.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:24:17.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:24:17.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:24:17.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:24:17.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:24:17.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:24:17.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:24:17.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-15T13:24:17.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-15T13:24:17.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-05-15T13:24:17.320 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-05-15T13:24:17.321 INFO:tasks.workunit.client.0.smithi005.stderr:start osd.3 2024-05-15T13:24:17.321 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:24:17.321 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-05-15T13:24:17.322 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-05-15T13:24:17.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-15T13:24:17.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-15T13:24:17.324 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-15T13:24:17.362 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:24:17.362+0000 7f6ed350c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:24:17.365 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:24:17.362+0000 7f6ed350c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:24:17.367 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:24:17.366+0000 7f6ed350c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:24:17.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-05-15T13:24:17.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:24:17.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-15T13:24:17.745 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:24:17.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:24:17.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:24:17.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:24:17.746 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T13:24:17.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:24:17.746 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:24:17.907 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:24:17.906+0000 7f6ed350c7c0 -1 Falling back to public interface 2024-05-15T13:24:18.181 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:24:19.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:24:19.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:24:19.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:24:19.183 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:24:19.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:24:19.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:24:19.610 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:24:20.519 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:24:20.518+0000 7f6ed350c7c0 -1 osd.3 84 log_to_monitors true 2024-05-15T13:24:20.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:24:20.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:24:20.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:24:20.613 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:24:20.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:24:20.613 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:24:21.074 INFO:tasks.workunit.client.0.smithi005.stderr:osd.3 up in weight 1 up_from 88 up_thru 82 down_at 85 last_clean_interval [82,84) [v2:127.0.0.1:6826/18365079,v1:127.0.0.1:6827/18365079] [v2:127.0.0.1:6828/18365079,v1:127.0.0.1:6829/18365079] exists,up 88fdbdeb-1be5-4922-9a2d-6149a04a85f6 2024-05-15T13:24:21.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:24:21.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:24:21.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:24:21.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-15T13:24:21.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:24:21.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:24:21.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:24:21.075 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:24:21.076 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:24:21.076 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:24:21.077 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:24:21.077 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:24:21.077 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:24:21.176 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:24:21.177 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:24:21.177 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:24:21.177 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:24:21.177 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:24:21.177 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:24:21.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:24:21.695 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:24:21.696 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:24:21.696 INFO:tasks.workunit.client.0.smithi005.stderr:3' 2024-05-15T13:24:21.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:24:21.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:24:21.696 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:24:21.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803841 2024-05-15T13:24:21.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803841 2024-05-15T13:24:21.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803841' 2024-05-15T13:24:21.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:24:21.912 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:24:22.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645325 2024-05-15T13:24:22.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645325 2024-05-15T13:24:22.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803841 1-309237645325' 2024-05-15T13:24:22.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:24:22.081 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:24:22.437 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378672 2024-05-15T13:24:22.437 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378672 2024-05-15T13:24:22.437 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803841 1-309237645325 2-81604378672' 2024-05-15T13:24:22.437 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:24:22.438 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:24:22.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122051 2024-05-15T13:24:22.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122051 2024-05-15T13:24:22.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803841 1-309237645325 2-81604378672 3-377957122051' 2024-05-15T13:24:22.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:24:22.603 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803841 2024-05-15T13:24:22.604 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:24:22.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:24:22.605 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803841 2024-05-15T13:24:22.605 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:24:22.606 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803841 2024-05-15T13:24:22.607 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803841' 2024-05-15T13:24:22.607 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.0 seq 25769803841 2024-05-15T13:24:22.607 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:24:23.064 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803839 -lt 25769803841 2024-05-15T13:24:23.064 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-15T13:24:24.066 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-15T13:24:24.067 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:24:24.490 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803841 -lt 25769803841 2024-05-15T13:24:24.490 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:24:24.491 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-309237645325 2024-05-15T13:24:24.491 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:24:24.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:24:24.493 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-309237645325 2024-05-15T13:24:24.493 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:24:24.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645325 2024-05-15T13:24:24.494 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 309237645325' 2024-05-15T13:24:24.494 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.1 seq 309237645325 2024-05-15T13:24:24.494 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:24:24.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645325 -lt 309237645325 2024-05-15T13:24:24.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:24:24.917 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378672 2024-05-15T13:24:24.917 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:24:24.918 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:24:24.919 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378672 2024-05-15T13:24:24.919 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:24:24.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378672 2024-05-15T13:24:24.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378672' 2024-05-15T13:24:24.921 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.2 seq 81604378672 2024-05-15T13:24:24.921 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:24:25.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378672 -lt 81604378672 2024-05-15T13:24:25.335 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:24:25.336 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-377957122051 2024-05-15T13:24:25.336 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:24:25.338 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:24:25.339 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-377957122051 2024-05-15T13:24:25.339 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:24:25.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122051 2024-05-15T13:24:25.340 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 377957122051' 2024-05-15T13:24:25.340 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.3 seq 377957122051 2024-05-15T13:24:25.341 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:24:25.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122051 -lt 377957122051 2024-05-15T13:24:25.801 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:24:25.802 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:24:25.802 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:24:26.368 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:24:26.368 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:24:26.369 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:24:26.369 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:24:26.369 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:24:26.369 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:24:26.369 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:24:26.369 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:24:26.771 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-15T13:24:26.772 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:24:26.772 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:24:26.772 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:24:27.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-05-15T13:24:27.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-05-15T13:24:27.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-05-15T13:24:27.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-05-15T13:24:27.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-15T13:24:27.310 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-05-15T13:24:27.410 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-15T13:24:27.410 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:24:27.411 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:24:27.411 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:24:27.411 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:24:27.411 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:24:27.412 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:24:27.412 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:24:27.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:24:27.905 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:24:27.905 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:24:27.906 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:24:28.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:24:28.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:24:28.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:24:28.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-05-15T13:24:28.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-05-15T13:24:29.040 INFO:tasks.workunit.client.0.smithi005.stderr:noout is unset 2024-05-15T13:24:29.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-80702-0-256 2024-05-15T13:24:29.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-15T13:24:29.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-15T13:24:29.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-80702-0-256 2024-05-15T13:24:29.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-15T13:24:29.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-15T13:24:29.061 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-80702-0-256 td/test-erasure-eio/COPY 2024-05-15T13:24:29.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-15T13:24:29.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-15T13:24:29.267 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-05-15T13:24:29.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-05-15T13:24:29.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-80702-0-256 td/test-erasure-eio 1 256 add 2024-05-15T13:24:29.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-80702-0-256 2024-05-15T13:24:29.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-05-15T13:24:29.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-05-15T13:24:29.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-05-15T13:24:29.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-05-15T13:24:29.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-05-15T13:24:29.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-05-15T13:24:29.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-05-15T13:24:29.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-05-15T13:24:29.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-05-15T13:24:29.270 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-80702-0-256 2024-05-15T13:24:29.271 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T13:24:29.271 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-80702-0-256 2024-05-15T13:24:29.271 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-80702-0-256 2024-05-15T13:24:29.271 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T13:24:29.719 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T13:24:29.720 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:24:29.720 INFO:tasks.workunit.client.0.smithi005.stderr:2' 2024-05-15T13:24:29.720 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-15T13:24:29.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-05-15T13:24:29.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-05-15T13:24:29.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-05-15T13:24:29.720 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-05-15T13:24:30.437 INFO:tasks.workunit.client.0.smithi005.stderr:noout is set 2024-05-15T13:24:30.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-05-15T13:24:30.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-80702-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:24:30.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-15T13:24:30.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-15T13:24:30.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-05-15T13:24:30.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-15T13:24:30.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-80702-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:24:30.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-15T13:24:30.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-15T13:24:30.461 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-05-15T13:24:30.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-15T13:24:30.462 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-05-15T13:24:30.462 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:24:30.462 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:24:30.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:24:30.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:24:30.463 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:24:30.774 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:24:30.774 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-80702-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:24:30.774 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-15T13:24:30.774 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-15T13:24:30.774 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-05-15T13:24:30.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-15T13:24:30.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-05-15T13:24:30.775 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-80702-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:24:33.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-05-15T13:24:33.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-15T13:24:33.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-15T13:24:33.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-05-15T13:24:33.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-15T13:24:33.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-05-15T13:24:33.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:24:33.357 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:24:33.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:24:33.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:24:33.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-15T13:24:33.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-15T13:24:33.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-15T13:24:33.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-15T13:24:33.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:24:33.358 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-15T13:24:33.358 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:24:33.358 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:24:33.359 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:24:33.360 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:24:33.360 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:24:33.360 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:24:33.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:24:33.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:24:33.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:24:33.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:24:33.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:24:33.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:24:33.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:24:33.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:24:33.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-15T13:24:33.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-15T13:24:33.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-05-15T13:24:33.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-05-15T13:24:33.361 INFO:tasks.workunit.client.0.smithi005.stderr:start osd.1 2024-05-15T13:24:33.361 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:24:33.362 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-05-15T13:24:33.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-05-15T13:24:33.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-15T13:24:33.363 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-15T13:24:33.364 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-15T13:24:33.402 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:24:33.402+0000 7f945c58c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:24:33.405 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:24:33.402+0000 7f945c58c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:24:33.407 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:24:33.406+0000 7f945c58c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:24:33.789 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-05-15T13:24:33.790 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:24:33.790 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-15T13:24:33.790 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:24:33.790 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:24:33.790 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:24:33.790 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:24:33.790 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T13:24:33.790 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:24:33.790 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:24:33.955 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:24:33.954+0000 7f945c58c7c0 -1 Falling back to public interface 2024-05-15T13:24:34.232 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:24:35.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:24:35.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:24:35.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:24:35.235 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:24:35.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:24:35.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:24:35.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:24:36.576 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:24:36.574+0000 7f945c58c7c0 -1 osd.1 93 log_to_monitors true 2024-05-15T13:24:36.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:24:36.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:24:36.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:24:36.667 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:24:36.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:24:36.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:24:37.244 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:24:38.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:24:38.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:24:38.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:24:38.246 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T13:24:38.247 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:24:38.247 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:24:38.956 INFO:tasks.workunit.client.0.smithi005.stderr:osd.1 up in weight 1 up_from 98 up_thru 98 down_at 94 last_clean_interval [72,93) [v2:127.0.0.1:6810/1796297717,v1:127.0.0.1:6811/1796297717] [v2:127.0.0.1:6812/1796297717,v1:127.0.0.1:6813/1796297717] exists,up 13fc1548-9ac0-4594-9e6f-fb2a2b817cbd 2024-05-15T13:24:38.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:24:38.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:24:38.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:24:38.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-15T13:24:38.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:24:38.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:24:38.956 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:24:38.957 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:24:38.958 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:24:38.958 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:24:38.958 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:24:38.958 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:24:38.958 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:24:39.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:24:39.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:24:39.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:24:39.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:24:39.084 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:24:39.084 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:24:39.524 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:24:39.524 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:24:39.524 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:24:39.524 INFO:tasks.workunit.client.0.smithi005.stderr:3' 2024-05-15T13:24:39.524 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:24:39.524 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:24:39.524 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:24:39.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803846 2024-05-15T13:24:39.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803846 2024-05-15T13:24:39.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803846' 2024-05-15T13:24:39.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:24:39.706 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:24:39.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795011 2024-05-15T13:24:39.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795011 2024-05-15T13:24:39.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803846 1-420906795011' 2024-05-15T13:24:39.859 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:24:39.859 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:24:40.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378678 2024-05-15T13:24:40.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378678 2024-05-15T13:24:40.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803846 1-420906795011 2-81604378678' 2024-05-15T13:24:40.008 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:24:40.008 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:24:40.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122056 2024-05-15T13:24:40.169 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122056 2024-05-15T13:24:40.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803846 1-420906795011 2-81604378678 3-377957122056' 2024-05-15T13:24:40.170 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:24:40.170 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803846 2024-05-15T13:24:40.170 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:24:40.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:24:40.171 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803846 2024-05-15T13:24:40.171 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:24:40.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803846 2024-05-15T13:24:40.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803846' 2024-05-15T13:24:40.173 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.0 seq 25769803846 2024-05-15T13:24:40.173 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:24:40.574 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803846 -lt 25769803846 2024-05-15T13:24:40.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:24:40.575 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-420906795011 2024-05-15T13:24:40.576 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:24:40.578 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:24:40.578 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-420906795011 2024-05-15T13:24:40.579 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:24:40.580 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795011 2024-05-15T13:24:40.580 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 420906795011' 2024-05-15T13:24:40.580 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.1 seq 420906795011 2024-05-15T13:24:40.580 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:24:41.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 420906795011 2024-05-15T13:24:41.003 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-15T13:24:42.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-15T13:24:42.004 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:24:42.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795011 -lt 420906795011 2024-05-15T13:24:42.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:24:42.430 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378678 2024-05-15T13:24:42.430 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:24:42.432 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:24:42.434 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:24:42.434 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378678 2024-05-15T13:24:42.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378678 2024-05-15T13:24:42.435 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378678' 2024-05-15T13:24:42.435 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.2 seq 81604378678 2024-05-15T13:24:42.435 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:24:42.865 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378678 -lt 81604378678 2024-05-15T13:24:42.866 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:24:42.866 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-377957122056 2024-05-15T13:24:42.866 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:24:42.869 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:24:42.870 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:24:42.870 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-377957122056 2024-05-15T13:24:42.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122056 2024-05-15T13:24:42.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 377957122056' 2024-05-15T13:24:42.871 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.3 seq 377957122056 2024-05-15T13:24:42.872 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:24:43.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122057 -lt 377957122056 2024-05-15T13:24:43.293 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:24:43.293 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:24:43.293 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:24:43.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:24:43.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:24:43.854 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:24:43.854 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:24:43.854 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:24:43.854 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:24:43.855 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:24:43.855 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:24:44.250 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:24:44.251 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:24:44.251 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:24:44.251 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:24:44.787 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:24:44.787 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:24:44.787 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:24:44.787 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-05-15T13:24:44.789 INFO:tasks.workunit.client.0.smithi005.stderr:1+0 records in 2024-05-15T13:24:44.789 INFO:tasks.workunit.client.0.smithi005.stderr:1+0 records out 2024-05-15T13:24:44.789 INFO:tasks.workunit.client.0.smithi005.stderr:256 bytes copied, 3.5296e-05 s, 7.3 MB/s 2024-05-15T13:24:44.789 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-80702-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:24:44.789 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-15T13:24:44.789 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-15T13:24:44.789 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-05-15T13:24:44.790 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-15T13:24:44.790 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-80702-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:24:44.790 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-15T13:24:44.790 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-15T13:24:44.790 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-05-15T13:24:44.790 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-15T13:24:44.790 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-05-15T13:24:44.790 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:24:44.790 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:24:44.790 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:24:44.790 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:24:44.790 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:24:45.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:24:45.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-80702-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:24:45.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-15T13:24:45.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-15T13:24:45.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-05-15T13:24:45.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-15T13:24:45.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-05-15T13:24:45.098 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-80702-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:24:47.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-05-15T13:24:47.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-15T13:24:47.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-15T13:24:47.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-05-15T13:24:47.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-15T13:24:47.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-05-15T13:24:47.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:24:47.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:24:47.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:24:47.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:24:47.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-15T13:24:47.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-15T13:24:47.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-15T13:24:47.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-15T13:24:47.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:24:47.666 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-15T13:24:47.666 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:24:47.666 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:24:47.666 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:24:47.666 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:24:47.667 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:24:47.667 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:24:47.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:24:47.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:24:47.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:24:47.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:24:47.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:24:47.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:24:47.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:24:47.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:24:47.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-15T13:24:47.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-15T13:24:47.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-05-15T13:24:47.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-05-15T13:24:47.669 INFO:tasks.workunit.client.0.smithi005.stderr:start osd.1 2024-05-15T13:24:47.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:24:47.669 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-05-15T13:24:47.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-05-15T13:24:47.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-15T13:24:47.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-15T13:24:47.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-15T13:24:47.703 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:24:47.701+0000 7fe2e183c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:24:47.706 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:24:47.705+0000 7fe2e183c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:24:47.707 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:24:47.705+0000 7fe2e183c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:24:48.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-05-15T13:24:48.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:24:48.128 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-15T13:24:48.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:24:48.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:24:48.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:24:48.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:24:48.129 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T13:24:48.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:24:48.129 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:24:48.259 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:24:48.257+0000 7fe2e183c7c0 -1 Falling back to public interface 2024-05-15T13:24:48.543 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:24:49.544 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:24:49.544 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:24:49.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:24:49.545 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:24:49.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:24:49.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:24:50.024 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:24:50.908 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:24:50.905+0000 7fe2e183c7c0 -1 osd.1 100 log_to_monitors true 2024-05-15T13:24:51.025 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:24:51.025 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:24:51.025 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:24:51.025 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:24:51.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:24:51.026 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:24:51.568 INFO:tasks.workunit.client.0.smithi005.stderr:osd.1 up in weight 1 up_from 105 up_thru 98 down_at 101 last_clean_interval [98,100) [v2:127.0.0.1:6810/414199732,v1:127.0.0.1:6811/414199732] [v2:127.0.0.1:6812/414199732,v1:127.0.0.1:6813/414199732] exists,up 13fc1548-9ac0-4594-9e6f-fb2a2b817cbd 2024-05-15T13:24:51.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:24:51.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:24:51.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:24:51.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-15T13:24:51.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:24:51.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:24:51.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:24:51.570 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:24:51.571 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:24:51.571 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:24:51.571 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:24:51.571 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:24:51.571 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:24:51.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:24:51.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:24:51.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:24:51.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:24:51.690 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:24:51.691 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:24:52.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:24:52.486 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:24:52.486 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:24:52.486 INFO:tasks.workunit.client.0.smithi005.stderr:3' 2024-05-15T13:24:52.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:24:52.486 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:24:52.487 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:24:52.747 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803851 2024-05-15T13:24:52.747 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803851 2024-05-15T13:24:52.747 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851' 2024-05-15T13:24:52.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:24:52.748 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:24:52.975 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566083 2024-05-15T13:24:52.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566083 2024-05-15T13:24:52.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851 1-450971566083' 2024-05-15T13:24:52.976 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:24:52.976 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:24:53.142 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378682 2024-05-15T13:24:53.142 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378682 2024-05-15T13:24:53.142 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851 1-450971566083 2-81604378682' 2024-05-15T13:24:53.142 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:24:53.142 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:24:53.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122061 2024-05-15T13:24:53.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122061 2024-05-15T13:24:53.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851 1-450971566083 2-81604378682 3-377957122061' 2024-05-15T13:24:53.299 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:24:53.299 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803851 2024-05-15T13:24:53.299 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:24:53.300 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:24:53.301 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803851 2024-05-15T13:24:53.301 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:24:53.302 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803851 2024-05-15T13:24:53.302 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803851' 2024-05-15T13:24:53.302 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.0 seq 25769803851 2024-05-15T13:24:53.303 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:24:53.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803849 -lt 25769803851 2024-05-15T13:24:53.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-15T13:24:54.730 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-15T13:24:54.731 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:24:55.204 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803851 -lt 25769803851 2024-05-15T13:24:55.204 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:24:55.205 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-450971566083 2024-05-15T13:24:55.205 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:24:55.207 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:24:55.208 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-450971566083 2024-05-15T13:24:55.208 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:24:55.209 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566083 2024-05-15T13:24:55.210 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 450971566083' 2024-05-15T13:24:55.210 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.1 seq 450971566083 2024-05-15T13:24:55.210 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:24:55.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566083 -lt 450971566083 2024-05-15T13:24:55.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:24:55.713 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378682 2024-05-15T13:24:55.713 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:24:55.715 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:24:55.716 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378682 2024-05-15T13:24:55.716 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:24:55.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378682 2024-05-15T13:24:55.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378682' 2024-05-15T13:24:55.718 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.2 seq 81604378682 2024-05-15T13:24:55.718 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:24:56.242 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378683 -lt 81604378682 2024-05-15T13:24:56.243 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:24:56.243 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-377957122061 2024-05-15T13:24:56.243 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:24:56.246 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:24:56.246 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-377957122061 2024-05-15T13:24:56.247 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:24:56.248 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122061 2024-05-15T13:24:56.248 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 377957122061' 2024-05-15T13:24:56.248 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.3 seq 377957122061 2024-05-15T13:24:56.248 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:24:56.697 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122061 -lt 377957122061 2024-05-15T13:24:56.697 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:24:56.698 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:24:56.698 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:24:57.259 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:24:57.260 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:24:57.260 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:24:57.261 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:24:57.261 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:24:57.261 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:24:57.261 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:24:57.261 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:24:57.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-15T13:24:57.668 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:24:57.668 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:24:57.669 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:24:58.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-05-15T13:24:58.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-05-15T13:24:58.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-05-15T13:24:58.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-05-15T13:24:58.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-15T13:24:58.261 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-05-15T13:24:58.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-15T13:24:58.362 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:24:58.364 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:24:58.364 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:24:58.364 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:24:58.365 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:24:58.365 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:24:58.365 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:24:58.772 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:24:58.773 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:24:58.773 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:24:58.773 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:24:59.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:24:59.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:24:59.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:24:59.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-05-15T13:24:59.328 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-05-15T13:25:00.091 INFO:tasks.workunit.client.0.smithi005.stderr:noout is unset 2024-05-15T13:25:00.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-80702-0-256 fail 2024-05-15T13:25:00.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-15T13:25:00.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-15T13:25:00.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-80702-0-256 2024-05-15T13:25:00.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-05-15T13:25:00.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-05-15T13:25:00.116 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-80702-0-256 td/test-erasure-eio/COPY 2024-05-15T13:25:00.283 INFO:tasks.workunit.client.0.smithi005.stderr:error getting pool-jerasure/obj-size-80702-0-256: (5) Input/output error 2024-05-15T13:25:00.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-05-15T13:25:00.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-05-15T13:25:00.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:312: TEST_rados_get_bad_size_shard_0: delete_erasure_coded_pool pool-jerasure 2024-05-15T13:25:00.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-15T13:25:00.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-15T13:25:00.902 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-jerasure' does not exist 2024-05-15T13:25:00.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-15T13:25:01.530 INFO:tasks.workunit.client.0.smithi005.stderr:erasure-code-profile myprofile does not exist 2024-05-15T13:25:01.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-15T13:25:01.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-15T13:25:01.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T13:25:01.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-15T13:25:01.550 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:25:01.550 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:25:01.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:25:01.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:25:01.551 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:25:01.663 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:25:01.664 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T13:25:01.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T13:25:01.665 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T13:25:01.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T13:25:01.666 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T13:25:01.666 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T13:25:01.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:25:01.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T13:25:01.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T13:25:01.668 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:25:01.668 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:25:01.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T13:25:01.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-15T13:25:01.701 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T13:25:01.701 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:25:01.702 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:25:01.702 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80702 2024-05-15T13:25:01.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T13:25:01.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T13:25:01.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-15T13:25:01.703 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-15T13:25:01.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-15T13:25:01.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-15T13:25:01.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-15T13:25:01.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-15T13:25:01.704 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-15T13:25:01.704 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:25:01.705 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:25:01.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:25:01.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:25:01.705 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:25:01.706 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:25:01.707 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T13:25:01.707 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T13:25:01.708 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T13:25:01.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T13:25:01.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T13:25:01.709 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T13:25:01.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:25:01.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T13:25:01.710 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T13:25:01.710 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:25:01.711 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:25:01.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-15T13:25:01.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-15T13:25:01.713 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T13:25:01.713 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:25:01.713 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:25:01.713 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80702 2024-05-15T13:25:01.714 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T13:25:01.714 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T13:25:01.714 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-15T13:25:01.716 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-15T13:25:01.716 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:25:01.716 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:25:01.716 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.80702 2024-05-15T13:25:01.718 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-15T13:25:01.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-15T13:25:01.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-15T13:25:01.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-15T13:25:01.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-15T13:25:01.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-15T13:25:01.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-15T13:25:01.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-15T13:25:01.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-15T13:25:01.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-15T13:25:01.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-15T13:25:02.108 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-15T13:25:02.108 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:25:02.108 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:25:02.109 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:25:02.109 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:25:02.109 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:25:02.109 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:25:02.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-15T13:25:02.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-15T13:25:02.294 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-15T13:25:02.294 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T13:25:02.295 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T13:25:02.295 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-15T13:25:02.296 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-15T13:25:02.296 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T13:25:02.296 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T13:25:02.296 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T13:25:02.296 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:25:02.296 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:25:02.296 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:25:02.296 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T13:25:02.297 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T13:25:02.297 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.80702/ceph-mon.a.asok config get fsid 2024-05-15T13:25:02.416 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-15T13:25:02.416 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-15T13:25:02.416 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-15T13:25:02.416 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-15T13:25:02.417 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-15T13:25:02.417 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-15T13:25:02.417 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T13:25:02.417 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T13:25:02.418 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:25:02.418 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:25:02.418 INFO:tasks.workunit.client.0.smithi005.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:25:02.418 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T13:25:02.418 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-15T13:25:02.418 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.80702/ceph-mon.a.asok config get mon_host 2024-05-15T13:25:02.530 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-15T13:25:02.530 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-15T13:25:02.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-15T13:25:02.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-15T13:25:02.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-15T13:25:02.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-15T13:25:02.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-15T13:25:03.425 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-15T13:25:03.426 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:25:03.426 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:25:03.426 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:25:03.426 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:25:03.426 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:25:03.426 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:25:03.427 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T13:25:03.429 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-15T13:25:03.464 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-15T13:25:03.465 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-15T13:25:03.764 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd' already exists 2024-05-15T13:25:03.781 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T13:25:04.784 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-15T13:25:04.784 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-15T13:25:04.784 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-15T13:25:04.784 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:25:04.784 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:25:04.784 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:25:04.785 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-mon.a.asok 2024-05-15T13:25:04.785 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-15T13:25:04.785 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-mon.a.asok log flush 2024-05-15T13:25:04.896 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T13:25:04.909 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-15T13:25:04.911 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T13:25:02.153+0000 7eff21b74c80 0 load: jerasure load: lrc 2024-05-15T13:25:04.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_1 td/test-erasure-eio 2024-05-15T13:25:04.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:316: TEST_rados_get_bad_size_shard_1: local dir=td/test-erasure-eio 2024-05-15T13:25:04.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:317: TEST_rados_get_bad_size_shard_1: setup_osds 4 2024-05-15T13:25:04.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-05-15T13:25:04.912 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-15T13:25:04.912 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-05-15T13:25:04.914 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-05-15T13:25:04.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:25:04.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-15T13:25:04.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:25:04.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:25:04.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-15T13:25:04.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:25:04.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-15T13:25:04.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:25:04.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:25:04.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:25:04.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:25:04.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-15T13:25:04.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-15T13:25:04.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:25:04.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:25:04.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:25:04.917 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:25:04.917 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:25:04.917 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:25:04.918 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:25:04.918 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:25:04.918 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:25:04.919 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:25:04.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:25:04.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:25:04.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:25:04.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:25:04.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:25:04.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:25:04.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:25:04.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:25:04.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:25:04.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:25:04.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:25:04.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:25:04.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-15T13:25:04.920 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:25:04.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7ba85b33-e16d-4f53-83f8-ca4b22054f6d 2024-05-15T13:25:04.920 INFO:tasks.workunit.client.0.smithi005.stdout:add osd0 7ba85b33-e16d-4f53-83f8-ca4b22054f6d 2024-05-15T13:25:04.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7ba85b33-e16d-4f53-83f8-ca4b22054f6d' 2024-05-15T13:25:04.921 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:25:04.936 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCwt0RmnK3oNxAAzj89ur9s9CibpCnVChtPcQ== 2024-05-15T13:25:04.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCwt0RmnK3oNxAAzj89ur9s9CibpCnVChtPcQ=="}' 2024-05-15T13:25:04.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7ba85b33-e16d-4f53-83f8-ca4b22054f6d -i td/test-erasure-eio/0/new.json 2024-05-15T13:25:05.248 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:25:05.264 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-15T13:25:05.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCwt0RmnK3oNxAAzj89ur9s9CibpCnVChtPcQ== --osd-uuid 7ba85b33-e16d-4f53-83f8-ca4b22054f6d 2024-05-15T13:25:05.298 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:05.297+0000 7f6e515c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:25:05.316 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:05.313+0000 7f6e515c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:25:05.318 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:05.317+0000 7f6e515c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:25:05.318 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:05.317+0000 7f6e515c07c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T13:25:05.318 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:05.317+0000 7f6e515c07c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T13:25:05.318 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:05.317+0000 7f6e515c07c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-15T13:25:05.356 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:05.353+0000 7f6e515c07c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-15T13:25:10.608 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-15T13:25:10.608 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:25:10.610 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-15T13:25:10.610 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd0 key to auth repository 2024-05-15T13:25:10.610 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:25:11.165 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.0 2024-05-15T13:25:11.166 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-15T13:25:11.166 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:25:11.166 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:25:11.166 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:25:11.166 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:25:11.198 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:11.193+0000 7f00a03637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:25:11.220 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:11.217+0000 7f00a03637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:25:11.222 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:11.221+0000 7f00a03637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:25:11.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-15T13:25:11.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:25:11.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-15T13:25:11.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:25:11.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:25:11.592 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:11.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:25:11.593 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:25:11.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:11.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:25:11.771 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:11.769+0000 7f00a03637c0 -1 Falling back to public interface 2024-05-15T13:25:12.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:13.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:13.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:13.006 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:25:13.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:25:13.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:13.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:25:13.450 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:14.302 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:14.301+0000 7f00a03637c0 -1 osd.0 0 log_to_monitors true 2024-05-15T13:25:14.452 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:14.452 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:14.452 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:25:14.453 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:25:14.453 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:14.453 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:25:15.195 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:16.196 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:16.196 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:16.197 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:25:16.197 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:25:16.197 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:16.197 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:25:16.662 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:17.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:17.664 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:17.664 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:25:17.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:25:17.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:17.665 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:25:18.185 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:19.187 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:19.187 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:19.187 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:25:19.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:25:19.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:19.188 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:25:19.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:20.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:20.654 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:20.654 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:25:20.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:25:20.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:20.655 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:25:21.147 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:22.149 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:22.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:22.150 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:25:22.150 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:25:22.151 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:22.151 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:25:22.609 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:23.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:23.611 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:23.611 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:25:23.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:25:23.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:23.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:25:24.158 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:25.159 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:25.159 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:25.160 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:25:25.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:25:25.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:25.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:25:25.584 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:26.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:26.585 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:26.586 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:25:26.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:25:26.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:26.586 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:25:27.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:28.076 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:28.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:28.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:25:28.077 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:25:28.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:28.077 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:25:29.161 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:30.162 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:30.162 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:30.163 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:25:30.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:25:30.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:30.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:25:31.004 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:32.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:32.005 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:32.006 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:25:32.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:25:32.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:32.006 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:25:32.500 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:33.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:33.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:33.502 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:25:33.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:25:33.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:33.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:25:34.283 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:35.284 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:35.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:35.285 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:25:35.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:25:35.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:35.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:25:35.818 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:36.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:36.820 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:36.820 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:25:36.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:25:36.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:36.821 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-15T13:25:37.289 INFO:tasks.workunit.client.0.smithi005.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1336410931,v1:127.0.0.1:6803/1336410931] [v2:127.0.0.1:6804/1336410931,v1:127.0.0.1:6805/1336410931] exists,up 7ba85b33-e16d-4f53-83f8-ca4b22054f6d 2024-05-15T13:25:37.291 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:25:37.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:25:37.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:25:37.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:25:37.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-15T13:25:37.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:25:37.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:25:37.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-15T13:25:37.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:25:37.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-15T13:25:37.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:25:37.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:25:37.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:25:37.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:25:37.292 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-15T13:25:37.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-15T13:25:37.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:25:37.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:25:37.293 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:25:37.293 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:25:37.293 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:25:37.293 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:25:37.293 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:25:37.293 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:25:37.293 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:25:37.294 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:25:37.294 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:25:37.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:25:37.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:25:37.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:25:37.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:25:37.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:25:37.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:25:37.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:25:37.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:25:37.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:25:37.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:25:37.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:25:37.295 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-15T13:25:37.295 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:25:37.296 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=be0ac664-1634-43c2-a010-34c3e219c44c 2024-05-15T13:25:37.296 INFO:tasks.workunit.client.0.smithi005.stdout:add osd1 be0ac664-1634-43c2-a010-34c3e219c44c 2024-05-15T13:25:37.296 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 be0ac664-1634-43c2-a010-34c3e219c44c' 2024-05-15T13:25:37.296 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:25:37.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDRt0RmXPgGExAABNpwQqjtrY+k4fSloS19fg== 2024-05-15T13:25:37.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDRt0RmXPgGExAABNpwQqjtrY+k4fSloS19fg=="}' 2024-05-15T13:25:37.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new be0ac664-1634-43c2-a010-34c3e219c44c -i td/test-erasure-eio/1/new.json 2024-05-15T13:25:37.783 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:25:37.799 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-15T13:25:37.800 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDRt0RmXPgGExAABNpwQqjtrY+k4fSloS19fg== --osd-uuid be0ac664-1634-43c2-a010-34c3e219c44c 2024-05-15T13:25:37.837 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:37.833+0000 7f8d8cc497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:25:37.839 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:37.837+0000 7f8d8cc497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:25:37.840 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:37.837+0000 7f8d8cc497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:25:37.841 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:37.837+0000 7f8d8cc497c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T13:25:37.841 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:37.837+0000 7f8d8cc497c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T13:25:37.841 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:37.837+0000 7f8d8cc497c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-15T13:25:37.880 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:37.877+0000 7f8d8cc497c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-15T13:25:43.747 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-15T13:25:43.747 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:25:43.749 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd1 key to auth repository 2024-05-15T13:25:43.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-15T13:25:43.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:25:44.318 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.1 2024-05-15T13:25:44.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-15T13:25:44.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:25:44.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:25:44.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:25:44.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:25:44.353 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:44.348+0000 7fb4b20257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:25:44.360 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:44.356+0000 7fb4b20257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:25:44.361 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:44.360+0000 7fb4b20257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:25:44.723 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-15T13:25:44.723 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:25:44.723 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-15T13:25:44.723 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:25:44.723 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:25:44.723 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:44.724 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:25:44.724 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:25:44.724 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:44.724 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:25:44.911 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:44.908+0000 7fb4b20257c0 -1 Falling back to public interface 2024-05-15T13:25:45.153 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:46.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:46.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:46.155 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:25:46.155 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:25:46.156 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:46.156 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:25:46.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:47.432 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:25:47.428+0000 7fb4b20257c0 -1 osd.1 0 log_to_monitors true 2024-05-15T13:25:47.644 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:47.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:47.645 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:25:47.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:25:47.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:47.645 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:25:48.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:49.202 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:49.202 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:49.202 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:25:49.203 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:25:49.203 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:49.203 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:25:49.657 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:50.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:50.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:50.659 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:25:50.659 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:25:50.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:50.660 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:25:51.172 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:52.174 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:52.174 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:52.174 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:25:52.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:25:52.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:52.175 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:25:52.727 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:53.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:53.728 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:53.729 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:25:53.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:25:53.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:53.729 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:25:54.266 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:55.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:55.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:55.270 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:25:55.271 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:25:55.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:55.271 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:25:55.713 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:56.715 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:56.715 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:56.716 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:25:56.716 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:25:56.716 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:56.716 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:25:57.180 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:58.182 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:58.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:58.183 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:25:58.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:25:58.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:58.183 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:25:58.748 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:25:59.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:25:59.749 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:25:59.750 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:25:59.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:25:59.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:25:59.750 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:26:00.618 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:01.619 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:01.620 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:01.620 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:26:01.620 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:26:01.620 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:01.621 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:26:02.316 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:03.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:03.317 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:03.318 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:26:03.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:26:03.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:03.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:26:03.958 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:04.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:04.960 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:04.960 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:26:04.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:26:04.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:04.961 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:26:05.823 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:06.825 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:06.825 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:06.825 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:26:06.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:26:06.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:06.826 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:26:07.578 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:08.580 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:08.580 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:08.580 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:26:08.581 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:26:08.581 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:08.581 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:26:09.323 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:10.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:10.325 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:10.325 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:26:10.326 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:26:10.326 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:10.326 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:26:10.825 INFO:tasks.workunit.client.0.smithi005.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/57370192,v1:127.0.0.1:6811/57370192] [v2:127.0.0.1:6812/57370192,v1:127.0.0.1:6813/57370192] exists,up be0ac664-1634-43c2-a010-34c3e219c44c 2024-05-15T13:26:10.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:26:10.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:26:10.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:26:10.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:26:10.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-15T13:26:10.827 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:26:10.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:26:10.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-15T13:26:10.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:26:10.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-15T13:26:10.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:26:10.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:26:10.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:26:10.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:26:10.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-15T13:26:10.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-15T13:26:10.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:26:10.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:26:10.828 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:26:10.828 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:26:10.828 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:26:10.829 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:26:10.829 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:26:10.829 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:26:10.829 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:26:10.829 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:26:10.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:26:10.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:26:10.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:26:10.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:26:10.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:26:10.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:26:10.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:26:10.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:26:10.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:26:10.829 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:26:10.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:26:10.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:26:10.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-15T13:26:10.830 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:26:10.830 INFO:tasks.workunit.client.0.smithi005.stdout:add osd2 492f064a-29ae-4517-8c26-2b48d108d68b 2024-05-15T13:26:10.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=492f064a-29ae-4517-8c26-2b48d108d68b 2024-05-15T13:26:10.830 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 492f064a-29ae-4517-8c26-2b48d108d68b' 2024-05-15T13:26:10.830 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:26:10.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDyt0RmAh5YMhAAzt1so2JykMkqroORCdPMDg== 2024-05-15T13:26:10.843 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDyt0RmAh5YMhAAzt1so2JykMkqroORCdPMDg=="}' 2024-05-15T13:26:10.854 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 492f064a-29ae-4517-8c26-2b48d108d68b -i td/test-erasure-eio/2/new.json 2024-05-15T13:26:11.384 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:26:11.403 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-15T13:26:11.405 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDyt0RmAh5YMhAAzt1so2JykMkqroORCdPMDg== --osd-uuid 492f064a-29ae-4517-8c26-2b48d108d68b 2024-05-15T13:26:11.440 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:11.436+0000 7fa54f36e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:26:11.487 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:11.484+0000 7fa54f36e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:26:11.488 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:11.484+0000 7fa54f36e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:26:11.489 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:11.484+0000 7fa54f36e7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T13:26:11.489 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:11.484+0000 7fa54f36e7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T13:26:11.489 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:11.484+0000 7fa54f36e7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-15T13:26:11.534 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:11.528+0000 7fa54f36e7c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-15T13:26:17.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-15T13:26:17.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:26:17.653 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd2 key to auth repository 2024-05-15T13:26:17.653 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-15T13:26:17.653 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:26:18.234 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.2 2024-05-15T13:26:18.234 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-15T13:26:18.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:26:18.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:26:18.235 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:26:18.236 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:26:18.269 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:18.264+0000 7f182187a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:26:18.282 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:18.280+0000 7f182187a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:26:18.283 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:18.280+0000 7f182187a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:26:18.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-15T13:26:18.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:26:18.695 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-15T13:26:18.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:26:18.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:26:18.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:18.696 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:26:18.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:26:18.696 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:18.697 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:26:18.867 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:18.864+0000 7f182187a7c0 -1 Falling back to public interface 2024-05-15T13:26:19.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:20.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:20.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:20.112 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:26:20.112 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:26:20.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:20.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:26:20.547 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:21.361 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:21.356+0000 7f182187a7c0 -1 osd.2 0 log_to_monitors true 2024-05-15T13:26:21.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:21.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:21.549 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:26:21.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:26:21.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:21.550 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:26:22.047 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:23.049 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:23.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:23.050 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:26:23.050 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:26:23.051 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:23.051 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:26:23.490 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:24.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:24.492 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:24.493 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:26:24.493 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:26:24.493 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:24.493 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:26:24.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:25.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:25.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:25.921 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:26:25.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:26:25.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:25.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:26:26.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:27.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:27.548 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:27.549 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:26:27.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:26:27.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:27.549 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:26:27.984 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:28.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:28.986 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:28.986 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:26:28.987 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:26:28.987 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:28.987 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:26:29.501 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:30.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:30.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:30.503 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:26:30.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:26:30.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:30.504 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:26:30.937 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:31.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:31.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:31.940 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:26:31.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:26:31.940 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:31.941 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:26:32.502 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:33.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:33.505 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:33.505 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:26:33.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:26:33.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:33.506 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:26:34.040 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:35.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:35.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:35.042 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:26:35.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:26:35.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:35.043 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:26:36.067 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:37.069 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:37.069 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:37.070 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:26:37.070 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:26:37.070 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:37.071 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:26:37.803 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:38.804 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:38.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:38.805 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:26:38.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:26:38.805 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:38.806 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:26:39.754 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:40.755 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:40.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:40.756 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:26:40.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:26:40.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:40.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:26:41.861 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:42.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:42.863 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:42.863 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:26:42.864 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:26:42.864 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:42.864 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:26:43.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:44.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:44.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:44.331 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:26:44.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:26:44.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:44.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:26:44.808 INFO:tasks.workunit.client.0.smithi005.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1269633787,v1:127.0.0.1:6819/1269633787] [v2:127.0.0.1:6820/1269633787,v1:127.0.0.1:6821/1269633787] exists,up 492f064a-29ae-4517-8c26-2b48d108d68b 2024-05-15T13:26:44.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:26:44.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:26:44.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:26:44.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-15T13:26:44.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-15T13:26:44.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-15T13:26:44.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-15T13:26:44.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-15T13:26:44.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-15T13:26:44.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-15T13:26:44.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:26:44.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:26:44.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:26:44.812 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:26:44.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-15T13:26:44.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-15T13:26:44.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-15T13:26:44.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-15T13:26:44.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:26:44.813 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-15T13:26:44.813 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:26:44.813 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:26:44.813 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:26:44.813 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:26:44.813 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:26:44.813 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:26:44.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:26:44.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:26:44.813 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-15T13:26:44.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-15T13:26:44.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:26:44.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:26:44.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:26:44.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:26:44.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:26:44.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:26:44.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-15T13:26:44.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-15T13:26:44.814 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-15T13:26:44.815 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-15T13:26:44.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=492ec86e-671a-4dab-9d36-a073749eb181 2024-05-15T13:26:44.815 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 492ec86e-671a-4dab-9d36-a073749eb181' 2024-05-15T13:26:44.815 INFO:tasks.workunit.client.0.smithi005.stdout:add osd3 492ec86e-671a-4dab-9d36-a073749eb181 2024-05-15T13:26:44.815 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-15T13:26:44.838 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAUuERm9xMDMhAA6t99cJa2JBdISumGTySFhQ== 2024-05-15T13:26:44.838 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAUuERm9xMDMhAA6t99cJa2JBdISumGTySFhQ=="}' 2024-05-15T13:26:44.838 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 492ec86e-671a-4dab-9d36-a073749eb181 -i td/test-erasure-eio/3/new.json 2024-05-15T13:26:45.341 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:26:45.359 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-15T13:26:45.360 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAUuERm9xMDMhAA6t99cJa2JBdISumGTySFhQ== --osd-uuid 492ec86e-671a-4dab-9d36-a073749eb181 2024-05-15T13:26:45.396 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:45.395+0000 7f57194ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:26:45.489 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:45.487+0000 7f57194ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:26:45.491 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:45.491+0000 7f57194ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:26:45.491 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:45.491+0000 7f57194ad7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T13:26:45.491 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:45.491+0000 7f57194ad7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T13:26:45.491 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:45.491+0000 7f57194ad7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-15T13:26:45.578 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:45.575+0000 7f57194ad7c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-15T13:26:51.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-15T13:26:51.708 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-15T13:26:51.709 INFO:tasks.workunit.client.0.smithi005.stdout:adding osd3 key to auth repository 2024-05-15T13:26:51.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-15T13:26:51.709 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-15T13:26:52.272 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-15T13:26:52.272 INFO:tasks.workunit.client.0.smithi005.stdout:start osd.3 2024-05-15T13:26:52.273 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-15T13:26:52.274 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-15T13:26:52.274 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-15T13:26:52.274 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:26:52.305 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:52.303+0000 7f928fe1e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:26:52.318 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:52.319+0000 7f928fe1e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:26:52.320 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:52.319+0000 7f928fe1e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:26:52.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-15T13:26:52.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:26:52.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-15T13:26:52.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:26:52.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:26:52.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:52.769 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-05-15T13:26:52.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:26:52.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:52.770 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:26:52.911 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:52.911+0000 7f928fe1e7c0 -1 Falling back to public interface 2024-05-15T13:26:53.217 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:54.218 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:54.219 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:54.219 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-05-15T13:26:54.219 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:26:54.219 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:54.219 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:26:54.649 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:55.417 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:26:55.415+0000 7f928fe1e7c0 -1 osd.3 0 log_to_monitors true 2024-05-15T13:26:55.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:55.651 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:55.651 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-05-15T13:26:55.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:26:55.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:55.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:26:56.250 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:57.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:57.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:57.253 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-05-15T13:26:57.253 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:26:57.253 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:57.253 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:26:57.874 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:26:58.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:26:58.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:26:58.876 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-15T13:26:58.877 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-05-15T13:26:58.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:26:58.877 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:26:59.329 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:27:00.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:27:00.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:27:00.332 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-05-15T13:27:00.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-15T13:27:00.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:27:00.332 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:27:00.817 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:27:01.818 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:27:01.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:27:01.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-15T13:27:01.819 INFO:tasks.workunit.client.0.smithi005.stdout:6 2024-05-15T13:27:01.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:27:01.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:27:02.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:27:03.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:27:03.286 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:27:03.287 INFO:tasks.workunit.client.0.smithi005.stdout:7 2024-05-15T13:27:03.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-15T13:27:03.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:27:03.287 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:27:03.712 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:27:04.714 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:27:04.714 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:27:04.714 INFO:tasks.workunit.client.0.smithi005.stdout:8 2024-05-15T13:27:04.715 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-15T13:27:04.715 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:27:04.715 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:27:05.171 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:27:06.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:27:06.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:27:06.173 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-15T13:27:06.173 INFO:tasks.workunit.client.0.smithi005.stdout:9 2024-05-15T13:27:06.174 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:27:06.174 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:27:06.756 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:27:07.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:27:07.757 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:27:07.757 INFO:tasks.workunit.client.0.smithi005.stdout:10 2024-05-15T13:27:07.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-15T13:27:07.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:27:07.758 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:27:08.471 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:27:09.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:27:09.473 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:27:09.473 INFO:tasks.workunit.client.0.smithi005.stdout:11 2024-05-15T13:27:09.474 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-15T13:27:09.474 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:27:09.474 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:27:10.445 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:27:11.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:27:11.447 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:27:11.447 INFO:tasks.workunit.client.0.smithi005.stdout:12 2024-05-15T13:27:11.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-15T13:27:11.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:27:11.448 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:27:12.249 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:27:13.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:27:13.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:27:13.251 INFO:tasks.workunit.client.0.smithi005.stdout:13 2024-05-15T13:27:13.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-15T13:27:13.251 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:27:13.252 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:27:13.901 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:27:14.902 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:27:14.902 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:27:14.902 INFO:tasks.workunit.client.0.smithi005.stdout:14 2024-05-15T13:27:14.903 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-15T13:27:14.903 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:27:14.903 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:27:15.591 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:27:16.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:27:16.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:27:16.593 INFO:tasks.workunit.client.0.smithi005.stdout:15 2024-05-15T13:27:16.593 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-15T13:27:16.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:27:16.594 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:27:17.316 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:27:18.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:27:18.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:27:18.318 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-15T13:27:18.318 INFO:tasks.workunit.client.0.smithi005.stdout:16 2024-05-15T13:27:18.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:27:18.319 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-15T13:27:18.761 INFO:tasks.workunit.client.0.smithi005.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/138946907,v1:127.0.0.1:6827/138946907] [v2:127.0.0.1:6828/138946907,v1:127.0.0.1:6829/138946907] exists,up 492ec86e-671a-4dab-9d36-a073749eb181 2024-05-15T13:27:18.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:27:18.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:27:18.761 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:27:18.762 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-15T13:27:18.762 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-15T13:27:18.762 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-15T13:27:18.763 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-15T13:27:18.763 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:27:18.763 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:27:18.763 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.80702/ceph-osd.0.asok 2024-05-15T13:27:18.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-15T13:27:18.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.80702/ceph-osd.0.asok log flush 2024-05-15T13:27:18.871 INFO:tasks.workunit.client.0.smithi005.stdout:{} 2024-05-15T13:27:18.884 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-15T13:27:18.889 INFO:tasks.workunit.client.0.smithi005.stdout:2024-05-15T13:25:11.773+0000 7f00a03637c0 0 load: jerasure load: lrc 2024-05-15T13:27:18.889 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:319: TEST_rados_get_bad_size_shard_1: local poolname=pool-jerasure 2024-05-15T13:27:18.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:320: TEST_rados_get_bad_size_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-05-15T13:27:18.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-15T13:27:18.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-15T13:27:18.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-05-15T13:27:18.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-15T13:27:18.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-05-15T13:27:18.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-15T13:27:18.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-05-15T13:27:19.717 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-15T13:27:19.718 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-15T13:27:20.525 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'pool-jerasure' already exists 2024-05-15T13:27:20.542 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-15T13:27:21.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-15T13:27:21.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:27:21.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:27:21.545 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:27:21.546 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:27:21.547 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:27:21.547 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:27:21.547 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:27:21.547 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:27:21.547 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:27:21.683 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:27:21.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:27:21.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:27:21.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:27:21.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:27:21.684 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:27:22.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:27:22.201 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:27:22.201 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:27:22.201 INFO:tasks.workunit.client.0.smithi005.stderr:3' 2024-05-15T13:27:22.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:27:22.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:27:22.201 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:27:22.368 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803802 2024-05-15T13:27:22.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803802 2024-05-15T13:27:22.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803802' 2024-05-15T13:27:22.369 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:27:22.369 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:27:22.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574868 2024-05-15T13:27:22.536 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574868 2024-05-15T13:27:22.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803802 1-55834574868' 2024-05-15T13:27:22.537 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:27:22.537 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:27:22.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378638 2024-05-15T13:27:22.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378638 2024-05-15T13:27:22.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803802 1-55834574868 2-81604378638' 2024-05-15T13:27:22.738 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:27:22.739 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:27:22.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084294 2024-05-15T13:27:22.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084294 2024-05-15T13:27:22.890 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803802 1-55834574868 2-81604378638 3-120259084294' 2024-05-15T13:27:22.891 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:27:22.891 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803802 2024-05-15T13:27:22.891 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:27:22.892 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:27:22.892 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803802 2024-05-15T13:27:22.892 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:27:22.893 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803802 2024-05-15T13:27:22.894 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.0 seq 25769803802 2024-05-15T13:27:22.894 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803802' 2024-05-15T13:27:22.894 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:27:23.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803800 -lt 25769803802 2024-05-15T13:27:23.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-15T13:27:24.331 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-15T13:27:24.332 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:27:24.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803802 -lt 25769803802 2024-05-15T13:27:24.760 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:27:24.760 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574868 2024-05-15T13:27:24.761 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:27:24.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:27:24.763 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574868 2024-05-15T13:27:24.764 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:27:24.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574868 2024-05-15T13:27:24.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574868' 2024-05-15T13:27:24.766 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.1 seq 55834574868 2024-05-15T13:27:24.766 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:27:25.198 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574868 -lt 55834574868 2024-05-15T13:27:25.198 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:27:25.199 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378638 2024-05-15T13:27:25.199 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:27:25.200 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:27:25.201 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378638 2024-05-15T13:27:25.201 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:27:25.202 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378638 2024-05-15T13:27:25.202 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.2 seq 81604378638 2024-05-15T13:27:25.203 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378638' 2024-05-15T13:27:25.203 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:27:25.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378639 -lt 81604378638 2024-05-15T13:27:25.627 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:27:25.628 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084294 2024-05-15T13:27:25.628 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:27:25.629 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:27:25.631 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084294 2024-05-15T13:27:25.631 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:27:25.632 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084294 2024-05-15T13:27:25.632 INFO:tasks.workunit.client.0.smithi005.stdout:waiting osd.3 seq 120259084294 2024-05-15T13:27:25.633 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084294' 2024-05-15T13:27:25.633 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:27:26.042 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084295 -lt 120259084294 2024-05-15T13:27:26.042 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:27:26.043 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:27:26.043 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:27:26.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:27:26.612 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:27:26.612 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:27:26.613 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:27:26.613 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:27:26.613 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:27:26.613 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:27:26.613 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:27:27.030 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:27:27.031 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:27:27.031 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:27:27.031 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:27:27.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:27:27.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:27:27.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:27:27.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:322: TEST_rados_get_bad_size_shard_1: local shard_id=1 2024-05-15T13:27:27.569 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:323: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 10 2024-05-15T13:27:27.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-05-15T13:27:27.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-05-15T13:27:27.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=1 2024-05-15T13:27:27.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-05-15T13:27:27.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=10 2024-05-15T13:27:27.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-05-15T13:27:27.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-05-15T13:27:27.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-05-15T13:27:27.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-80702-1-10 2024-05-15T13:27:27.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-80702-1-10 2024-05-15T13:27:27.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-15T13:27:27.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-15T13:27:27.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-80702-1-10 2024-05-15T13:27:27.570 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:27:27.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-15T13:27:27.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:27:27.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-15T13:27:27.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:27:27.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-15T13:27:27.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-15T13:27:27.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-15T13:27:27.571 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-80702-1-10 td/test-erasure-eio/ORIGINAL 2024-05-15T13:27:27.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-80702-1-10 td/test-erasure-eio 1 10 set 2024-05-15T13:27:27.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-80702-1-10 2024-05-15T13:27:27.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-05-15T13:27:27.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-05-15T13:27:27.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-05-15T13:27:27.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-05-15T13:27:27.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-05-15T13:27:27.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-05-15T13:27:27.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-05-15T13:27:27.778 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-05-15T13:27:27.779 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-05-15T13:27:27.779 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-80702-1-10 2024-05-15T13:27:27.779 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T13:27:27.779 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-80702-1-10 2024-05-15T13:27:27.779 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-80702-1-10 2024-05-15T13:27:27.779 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T13:27:28.206 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T13:27:28.206 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:27:28.206 INFO:tasks.workunit.client.0.smithi005.stderr:2' 2024-05-15T13:27:28.206 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-15T13:27:28.206 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-05-15T13:27:28.206 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-05-15T13:27:28.206 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-05-15T13:27:28.206 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-05-15T13:27:28.859 INFO:tasks.workunit.client.0.smithi005.stderr:noout is set 2024-05-15T13:27:28.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-05-15T13:27:28.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-05-15T13:27:28.879 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-05-15T13:27:28.881 INFO:tasks.workunit.client.0.smithi005.stderr:1+0 records in 2024-05-15T13:27:28.881 INFO:tasks.workunit.client.0.smithi005.stderr:1+0 records out 2024-05-15T13:27:28.882 INFO:tasks.workunit.client.0.smithi005.stderr:10 bytes copied, 0.000190671 s, 52.4 kB/s 2024-05-15T13:27:28.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-80702-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:27:28.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-15T13:27:28.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-15T13:27:28.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-05-15T13:27:28.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-15T13:27:28.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-80702-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:27:28.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-15T13:27:28.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-15T13:27:28.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-05-15T13:27:28.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-15T13:27:28.882 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-05-15T13:27:28.882 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:27:28.883 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:27:28.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:27:28.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:27:28.883 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:27:29.192 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:27:29.192 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-80702-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:27:29.192 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-15T13:27:29.192 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-15T13:27:29.192 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-05-15T13:27:29.192 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-15T13:27:29.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-05-15T13:27:29.193 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-80702-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:27:31.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-05-15T13:27:31.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-15T13:27:31.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-15T13:27:31.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-05-15T13:27:31.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-15T13:27:31.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-05-15T13:27:31.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:27:31.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:27:31.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:27:31.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:27:31.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-15T13:27:31.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-15T13:27:31.669 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-15T13:27:31.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-15T13:27:31.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:27:31.670 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-15T13:27:31.670 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:27:31.670 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:27:31.671 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:27:31.671 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:27:31.671 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:27:31.671 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:27:31.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:27:31.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:27:31.671 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:27:31.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:27:31.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:27:31.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:27:31.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:27:31.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:27:31.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-15T13:27:31.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-15T13:27:31.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-05-15T13:27:31.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-05-15T13:27:31.672 INFO:tasks.workunit.client.0.smithi005.stderr:start osd.1 2024-05-15T13:27:31.673 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:27:31.673 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-05-15T13:27:31.674 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-05-15T13:27:31.674 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-15T13:27:31.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-15T13:27:31.675 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-15T13:27:31.713 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:27:31.711+0000 7fda128857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:27:31.716 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:27:31.715+0000 7fda128857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:27:31.717 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:27:31.715+0000 7fda128857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:27:32.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-05-15T13:27:32.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:27:32.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-15T13:27:32.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:27:32.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:27:32.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:27:32.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:27:32.110 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T13:27:32.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:27:32.110 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:27:32.293 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:27:32.291+0000 7fda128857c0 -1 Falling back to public interface 2024-05-15T13:27:32.553 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:27:33.555 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:27:33.556 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:27:33.556 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:27:33.556 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:27:33.556 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:27:33.556 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:27:34.010 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:27:35.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:27:35.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:27:35.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:27:35.012 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:27:35.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:27:35.012 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:27:35.063 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:27:35.063+0000 7fda128857c0 -1 osd.1 37 log_to_monitors true 2024-05-15T13:27:35.466 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:27:36.467 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:27:36.467 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:27:36.468 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-15T13:27:36.468 INFO:tasks.workunit.client.0.smithi005.stderr:3 2024-05-15T13:27:36.468 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:27:36.468 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-15T13:27:37.100 INFO:tasks.workunit.client.0.smithi005.stderr:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 40 last_clean_interval [13,37) [v2:127.0.0.1:6810/979134310,v1:127.0.0.1:6811/979134310] [v2:127.0.0.1:6812/979134310,v1:127.0.0.1:6813/979134310] exists,up be0ac664-1634-43c2-a010-34c3e219c44c 2024-05-15T13:27:37.101 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:27:37.101 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:27:37.101 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:27:37.101 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-15T13:27:37.101 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:27:37.101 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:27:37.101 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:27:37.101 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:27:37.102 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:27:37.102 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:27:37.102 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:27:37.102 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:27:37.102 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:27:37.223 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:27:37.223 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:27:37.223 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:27:37.223 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:27:37.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:27:37.224 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:27:37.684 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:27:37.684 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:27:37.684 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:27:37.684 INFO:tasks.workunit.client.0.smithi005.stderr:3' 2024-05-15T13:27:37.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:27:37.685 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:27:37.685 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:27:37.871 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803807 2024-05-15T13:27:37.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803807 2024-05-15T13:27:37.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803807' 2024-05-15T13:27:37.872 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:27:37.872 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:27:38.036 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561027 2024-05-15T13:27:38.036 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561027 2024-05-15T13:27:38.036 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803807 1-188978561027' 2024-05-15T13:27:38.036 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:27:38.036 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:27:38.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378643 2024-05-15T13:27:38.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378643 2024-05-15T13:27:38.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803807 1-188978561027 2-81604378643' 2024-05-15T13:27:38.201 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:27:38.201 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:27:38.376 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084299 2024-05-15T13:27:38.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084299 2024-05-15T13:27:38.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803807 1-188978561027 2-81604378643 3-120259084299' 2024-05-15T13:27:38.377 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:27:38.377 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803807 2024-05-15T13:27:38.377 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:27:38.378 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:27:38.379 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803807 2024-05-15T13:27:38.380 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:27:38.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803807 2024-05-15T13:27:38.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803807' 2024-05-15T13:27:38.382 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.0 seq 25769803807 2024-05-15T13:27:38.382 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:27:38.817 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803807 -lt 25769803807 2024-05-15T13:27:38.817 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:27:38.818 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-188978561027 2024-05-15T13:27:38.818 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:27:38.819 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:27:38.820 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-188978561027 2024-05-15T13:27:38.820 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:27:38.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561027 2024-05-15T13:27:38.822 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 188978561027' 2024-05-15T13:27:38.822 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.1 seq 188978561027 2024-05-15T13:27:38.822 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:27:39.245 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561027 -lt 188978561027 2024-05-15T13:27:39.245 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:27:39.246 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:27:39.246 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378643 2024-05-15T13:27:39.247 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:27:39.248 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378643 2024-05-15T13:27:39.248 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:27:39.249 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378643 2024-05-15T13:27:39.249 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378643' 2024-05-15T13:27:39.249 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.2 seq 81604378643 2024-05-15T13:27:39.249 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:27:39.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378643 -lt 81604378643 2024-05-15T13:27:39.667 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:27:39.668 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084299 2024-05-15T13:27:39.668 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:27:39.670 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:27:39.671 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084299 2024-05-15T13:27:39.671 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:27:39.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084299 2024-05-15T13:27:39.672 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084299' 2024-05-15T13:27:39.672 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.3 seq 120259084299 2024-05-15T13:27:39.673 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:27:40.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084297 -lt 120259084299 2024-05-15T13:27:40.113 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-15T13:27:41.115 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-15T13:27:41.116 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:27:41.546 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084300 -lt 120259084299 2024-05-15T13:27:41.546 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:27:41.547 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:27:41.547 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:27:42.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:27:42.081 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:27:42.082 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:27:42.082 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:27:42.082 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:27:42.082 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:27:42.082 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:27:42.082 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:27:42.503 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:27:42.503 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:27:42.504 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:27:42.504 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:27:43.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:27:43.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:27:43.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:27:43.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-05-15T13:27:43.075 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-05-15T13:27:43.892 INFO:tasks.workunit.client.0.smithi005.stderr:noout is unset 2024-05-15T13:27:43.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-80702-1-10 2024-05-15T13:27:43.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-15T13:27:43.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-15T13:27:43.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-80702-1-10 2024-05-15T13:27:43.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-15T13:27:43.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-15T13:27:43.920 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-80702-1-10 td/test-erasure-eio/COPY 2024-05-15T13:27:44.089 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-15T13:27:44.090 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-15T13:27:44.093 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 1 + 1 2024-05-15T13:27:44.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=2 2024-05-15T13:27:44.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-80702-1-10 td/test-erasure-eio 2 10 set 2024-05-15T13:27:44.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-80702-1-10 2024-05-15T13:27:44.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-05-15T13:27:44.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-05-15T13:27:44.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-05-15T13:27:44.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=2 2024-05-15T13:27:44.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-05-15T13:27:44.095 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-05-15T13:27:44.096 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-05-15T13:27:44.096 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-05-15T13:27:44.096 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-05-15T13:27:44.096 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-80702-1-10 2024-05-15T13:27:44.096 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-15T13:27:44.096 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-80702-1-10 2024-05-15T13:27:44.097 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-15T13:27:44.097 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-80702-1-10 2024-05-15T13:27:44.519 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-15T13:27:44.519 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:27:44.519 INFO:tasks.workunit.client.0.smithi005.stderr:2' 2024-05-15T13:27:44.520 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-15T13:27:44.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-05-15T13:27:44.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-05-15T13:27:44.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=2 2024-05-15T13:27:44.520 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-05-15T13:27:45.264 INFO:tasks.workunit.client.0.smithi005.stderr:noout is set 2024-05-15T13:27:45.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-05-15T13:27:45.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-05-15T13:27:45.285 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-05-15T13:27:45.287 INFO:tasks.workunit.client.0.smithi005.stderr:1+0 records in 2024-05-15T13:27:45.287 INFO:tasks.workunit.client.0.smithi005.stderr:1+0 records out 2024-05-15T13:27:45.287 INFO:tasks.workunit.client.0.smithi005.stderr:10 bytes copied, 0.000183705 s, 54.4 kB/s 2024-05-15T13:27:45.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-80702-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:27:45.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-15T13:27:45.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-15T13:27:45.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-05-15T13:27:45.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-15T13:27:45.288 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-80702-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:27:45.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-15T13:27:45.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-15T13:27:45.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-05-15T13:27:45.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-15T13:27:45.289 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-05-15T13:27:45.290 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:27:45.290 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:27:45.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:27:45.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:27:45.290 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:27:45.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:27:45.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-80702-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:27:45.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-15T13:27:45.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-15T13:27:45.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-05-15T13:27:45.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-15T13:27:45.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-05-15T13:27:45.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-80702-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-15T13:27:47.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-05-15T13:27:47.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-15T13:27:47.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-15T13:27:47.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-05-15T13:27:47.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-15T13:27:47.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-05-15T13:27:47.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-15T13:27:47.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-15T13:27:47.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-15T13:27:47.763 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-15T13:27:47.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-15T13:27:47.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-15T13:27:47.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-15T13:27:47.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-15T13:27:47.764 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-15T13:27:47.764 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-15T13:27:47.764 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-15T13:27:47.764 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-15T13:27:47.764 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-15T13:27:47.764 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:27:47.764 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:27:47.764 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:27:47.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' 2024-05-15T13:27:47.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-15T13:27:47.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-15T13:27:47.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-15T13:27:47.765 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-15T13:27:47.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-15T13:27:47.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-15T13:27:47.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-15T13:27:47.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-15T13:27:47.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-15T13:27:47.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-05-15T13:27:47.766 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-05-15T13:27:47.767 INFO:tasks.workunit.client.0.smithi005.stderr:start osd.2 2024-05-15T13:27:47.767 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=6ce4df02-c1d5-4cc4-b85d-5c899ee50853 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.80702/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-15T13:27:47.767 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-05-15T13:27:47.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-05-15T13:27:47.768 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-15T13:27:47.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-15T13:27:47.769 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-15T13:27:47.801 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:27:47.798+0000 7ff9750687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:27:47.805 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:27:47.802+0000 7ff9750687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:27:47.806 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:27:47.806+0000 7ff9750687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-15T13:27:48.223 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-05-15T13:27:48.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-15T13:27:48.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-15T13:27:48.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-15T13:27:48.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-15T13:27:48.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:27:48.224 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-15T13:27:48.225 INFO:tasks.workunit.client.0.smithi005.stderr:0 2024-05-15T13:27:48.225 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:27:48.225 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:27:48.364 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:27:48.362+0000 7ff9750687c0 -1 Falling back to public interface 2024-05-15T13:27:48.650 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:27:49.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:27:49.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:27:49.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-15T13:27:49.652 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:27:49.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:27:49.652 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:27:50.070 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-15T13:27:51.072 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-15T13:27:51.072 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-15T13:27:51.072 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-15T13:27:51.072 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:27:51.072 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-15T13:27:51.072 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-15T13:27:51.084 INFO:tasks.workunit.client.0.smithi005.stderr:2024-05-15T13:27:51.082+0000 7ff9750687c0 -1 osd.2 48 log_to_monitors true 2024-05-15T13:27:51.599 INFO:tasks.workunit.client.0.smithi005.stderr:osd.2 up in weight 1 up_from 54 up_thru 40 down_at 51 last_clean_interval [19,48) [v2:127.0.0.1:6818/1708312208,v1:127.0.0.1:6819/1708312208] [v2:127.0.0.1:6820/1708312208,v1:127.0.0.1:6821/1708312208] exists,up 492f064a-29ae-4517-8c26-2b48d108d68b 2024-05-15T13:27:51.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-15T13:27:51.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-15T13:27:51.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-15T13:27:51.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-15T13:27:51.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-15T13:27:51.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-15T13:27:51.599 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-15T13:27:51.599 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-15T13:27:51.600 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-15T13:27:51.600 INFO:tasks.workunit.client.0.smithi005.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-15T13:27:51.600 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-15T13:27:51.600 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-15T13:27:51.600 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-15T13:27:51.715 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-15T13:27:51.715 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-15T13:27:51.715 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-15T13:27:51.715 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-15T13:27:51.715 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-15T13:27:51.715 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-15T13:27:52.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-15T13:27:52.382 INFO:tasks.workunit.client.0.smithi005.stderr:1 2024-05-15T13:27:52.382 INFO:tasks.workunit.client.0.smithi005.stderr:2 2024-05-15T13:27:52.382 INFO:tasks.workunit.client.0.smithi005.stderr:3' 2024-05-15T13:27:52.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-15T13:27:52.382 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:27:52.383 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-15T13:27:52.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803812 2024-05-15T13:27:52.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803812 2024-05-15T13:27:52.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803812' 2024-05-15T13:27:52.643 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:27:52.643 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-15T13:27:52.846 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561032 2024-05-15T13:27:52.846 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561032 2024-05-15T13:27:52.846 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803812 1-188978561032' 2024-05-15T13:27:52.846 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:27:52.846 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-15T13:27:53.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928233987 2024-05-15T13:27:53.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928233987 2024-05-15T13:27:53.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803812 1-188978561032 2-231928233987' 2024-05-15T13:27:53.002 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-15T13:27:53.002 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-15T13:27:53.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084304 2024-05-15T13:27:53.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084304 2024-05-15T13:27:53.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803812 1-188978561032 2-231928233987 3-120259084304' 2024-05-15T13:27:53.160 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:27:53.160 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803812 2024-05-15T13:27:53.160 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:27:53.161 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-15T13:27:53.162 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803812 2024-05-15T13:27:53.162 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:27:53.163 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803812 2024-05-15T13:27:53.164 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803812' 2024-05-15T13:27:53.164 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.0 seq 25769803812 2024-05-15T13:27:53.164 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:27:53.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803810 -lt 25769803812 2024-05-15T13:27:53.575 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-15T13:27:54.576 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-15T13:27:54.577 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-15T13:27:55.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803812 -lt 25769803812 2024-05-15T13:27:55.027 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:27:55.027 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:27:55.028 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-188978561032 2024-05-15T13:27:55.029 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-15T13:27:55.030 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:27:55.030 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-188978561032 2024-05-15T13:27:55.031 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561032 2024-05-15T13:27:55.031 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 188978561032' 2024-05-15T13:27:55.031 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.1 seq 188978561032 2024-05-15T13:27:55.031 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-15T13:27:55.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561032 -lt 188978561032 2024-05-15T13:27:55.477 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:27:55.478 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-231928233987 2024-05-15T13:27:55.478 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:27:55.482 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-15T13:27:55.482 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:27:55.483 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-231928233987 2024-05-15T13:27:55.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928233987 2024-05-15T13:27:55.483 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 231928233987' 2024-05-15T13:27:55.483 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.2 seq 231928233987 2024-05-15T13:27:55.483 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-15T13:27:55.893 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928233987 -lt 231928233987 2024-05-15T13:27:55.893 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-15T13:27:55.894 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084304 2024-05-15T13:27:55.894 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-15T13:27:55.896 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-15T13:27:55.897 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084304 2024-05-15T13:27:55.897 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-15T13:27:55.898 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084304 2024-05-15T13:27:55.899 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084304' 2024-05-15T13:27:55.899 INFO:tasks.workunit.client.0.smithi005.stderr:waiting osd.3 seq 120259084304 2024-05-15T13:27:55.899 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-15T13:27:56.358 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084305 -lt 120259084304 2024-05-15T13:27:56.359 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-15T13:27:56.359 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:27:56.359 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:27:56.915 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-15T13:27:56.915 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-15T13:27:56.915 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-15T13:27:56.915 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-15T13:27:56.915 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-15T13:27:56.916 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-15T13:27:56.916 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-15T13:27:56.916 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-15T13:27:57.327 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-15T13:27:57.328 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-15T13:27:57.328 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-15T13:27:57.328 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-15T13:27:57.865 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-15T13:27:57.865 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-15T13:27:57.865 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-15T13:27:57.865 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-05-15T13:27:57.866 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-05-15T13:27:58.508 INFO:tasks.workunit.client.0.smithi005.stderr:noout is unset 2024-05-15T13:27:58.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-80702-1-10 fail 2024-05-15T13:27:58.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-15T13:27:58.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-15T13:27:58.531 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-80702-1-10 2024-05-15T13:27:58.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-05-15T13:27:58.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-05-15T13:27:58.532 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-80702-1-10 td/test-erasure-eio/COPY 2024-05-15T13:27:58.603 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-05-15T13:27:58.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: return 1 2024-05-15T13:27:58.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:323: TEST_rados_get_bad_size_shard_1: return 1 2024-05-15T13:27:58.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: return 1 2024-05-15T13:27:58.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: code=1 2024-05-15T13:27:58.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/test-erasure-eio 1 2024-05-15T13:27:58.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-15T13:27:58.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-05-15T13:27:58.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-15T13:27:58.604 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-15T13:27:58.604 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-15T13:27:58.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-15T13:27:58.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-15T13:27:58.604 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-15T13:27:58.916 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-15T13:27:58.916 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-15T13:27:58.917 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-15T13:27:58.918 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-15T13:27:58.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-15T13:27:58.919 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-15T13:27:58.920 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-15T13:27:58.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:27:58.921 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-15T13:27:58.922 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-15T13:27:58.922 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T13:27:58.923 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:27:58.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-05-15T13:27:58.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-05-15T13:27:58.925 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-05-15T13:27:58.926 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/test-erasure-eio/mgr.x.log td/test-erasure-eio/mon.a.log td/test-erasure-eio/osd.0.log td/test-erasure-eio/osd.1.log td/test-erasure-eio/osd.2.log td/test-erasure-eio/osd.3.log /home/ubuntu/cephtest/archive/log 2024-05-15T13:27:58.928 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-15T13:27:58.944 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-15T13:27:58.944 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-15T13:27:58.945 INFO:tasks.workunit.client.0.smithi005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.80702 2024-05-15T13:27:58.945 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.80702 2024-05-15T13:27:58.946 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-15T13:27:58.946 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-15T13:27:58.946 INFO:tasks.workunit.client.0.smithi005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 1 2024-05-15T13:27:58.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T13:27:58.947 INFO:tasks.workunit:Stopping ['erasure-code'] on client.0... 2024-05-15T13:27:58.947 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-15T13:27:59.280 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test erasure-code/test-erasure-eio.sh) on smithi005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae2481253f9b2fa3451954e615eedb67ebdd5314 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh' 2024-05-15T13:27:59.520 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c43d76d00002482d89e44aff3b00cd8d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test erasure-code/test-erasure-eio.sh) on smithi005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae2481253f9b2fa3451954e615eedb67ebdd5314 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh' 2024-05-15T13:27:59.523 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-15T13:27:59.531 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-15T13:27:59.531 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-15T13:27:59.574 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-15T13:27:59.584 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-15T13:27:59.584 DEBUG:teuthology.orchestra.run.smithi005:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T13:27:59.606 INFO:teuthology.orchestra.run.smithi005.stdout: remote refid st t when poll reach delay offset jitter 2024-05-15T13:27:59.606 INFO:teuthology.orchestra.run.smithi005.stdout:============================================================================== 2024-05-15T13:27:59.606 INFO:teuthology.orchestra.run.smithi005.stdout:*hv01.front.sepi 67.205.162.81 3 u 60 128 377 0.092 -2.724 0.323 2024-05-15T13:27:59.606 INFO:teuthology.orchestra.run.smithi005.stdout:+hv02.front.sepi 63.231.80.2 3 u 128 128 377 0.103 -0.170 0.304 2024-05-15T13:27:59.607 INFO:teuthology.orchestra.run.smithi005.stdout:+hv03.front.sepi 74.6.168.72 3 u 73 128 377 0.143 -0.643 0.329 2024-05-15T13:27:59.607 INFO:teuthology.orchestra.run.smithi005.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-05-15T13:27:59.607 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-15T13:27:59.616 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-15T13:27:59.616 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-15T13:27:59.624 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-15T13:27:59.632 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-15T13:27:59.640 INFO:teuthology.task.internal:Duration was 4511.460448 seconds 2024-05-15T13:27:59.640 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-15T13:27:59.648 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-15T13:27:59.649 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-15T13:27:59.684 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-15T13:27:59.684 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi005.front.sepia.ceph.com 2024-05-15T13:27:59.685 DEBUG:teuthology.orchestra.run.smithi005:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-15T13:27:59.734 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-15T13:27:59.735 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-15T13:27:59.817 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-15T13:27:59.818 DEBUG:teuthology.orchestra.run.smithi005:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-15T13:27:59.912 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-15T13:27:59.921 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-15T13:27:59.921 DEBUG:teuthology.orchestra.run.smithi005:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-15T13:27:59.932 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-15T13:27:59.941 DEBUG:teuthology.orchestra.run.smithi005:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:27:59.984 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = core 2024-05-15T13:28:00.010 DEBUG:teuthology.orchestra.run.smithi005:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:28:00.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T13:28:00.058 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-15T13:28:00.068 INFO:teuthology.task.internal:Transferring archived files... 2024-05-15T13:28:00.070 DEBUG:teuthology.misc:Transferring archived files from smithi005:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707093/remote/smithi005 2024-05-15T13:28:00.071 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-15T13:28:01.150 INFO:teuthology.task.internal:Removing archive directory... 2024-05-15T13:28:01.150 DEBUG:teuthology.orchestra.run.smithi005:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-15T13:28:01.185 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-15T13:28:01.195 INFO:teuthology.task.internal:Not uploading archives. 2024-05-15T13:28:01.195 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-15T13:28:01.290 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-15T13:28:01.291 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-15T13:28:01.298 INFO:teuthology.orchestra.run.smithi005.stdout: 658624 4 drwxr-xr-x 3 ubuntu ubuntu 4096 May 15 13:28 /home/ubuntu/cephtest 2024-05-15T13:28:01.298 INFO:teuthology.orchestra.run.smithi005.stdout: 671395 4 drwxrwxr-x 3 ubuntu ubuntu 4096 May 15 12:21 /home/ubuntu/cephtest/mnt.0 2024-05-15T13:28:01.299 INFO:teuthology.orchestra.run.smithi005.stdout: 673421 4 drwxrwxr-x 3 ubuntu ubuntu 4096 May 15 12:43 /home/ubuntu/cephtest/mnt.0/client.0 2024-05-15T13:28:01.299 INFO:teuthology.orchestra.run.smithi005.stdout: 688526 4 drwxrwxr-x 3 ubuntu ubuntu 4096 May 15 12:43 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-15T13:28:01.299 INFO:teuthology.orchestra.run.smithi005.stdout: 688527 4 drwxrwxr-x 2 ubuntu ubuntu 4096 May 15 13:27 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-05-15T13:28:01.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T13:28:01.300 INFO:teuthology.orchestra.run.smithi005.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-05-15T13:28:01.301 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi005 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-05-15T13:28:01.305 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-15T13:28:01.317 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-15T13:28:01.328 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi005 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-05-15T13:28:01.328 INFO:teuthology.nuke:Checking targets against current locks 2024-05-15T13:28:01.354 DEBUG:teuthology.nuke:shortname: smithi005 2024-05-15T13:28:01.355 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-15T13:28:01.379 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707093', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 12:03:56.773932', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGZ7WMWDQYIDxvgoIH/K16opOd3KZ8H0EO+PrsAUOhJzL6a4sE9K4vrUjm6StTzGeSwQ1UHQ7MYtR/PhaTPkKQ8='} 2024-05-15T13:28:01.430 INFO:teuthology.orchestra.console.smithi005:Power off 2024-05-15T13:28:01.430 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-15T13:28:01.454 DEBUG:teuthology.orchestra.console.smithi005:power off output: Chassis Power Control: Down/Off 2024-05-15T13:28:01.454 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:28:01.486 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is on 2024-05-15T13:28:05.487 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:28:05.596 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is on 2024-05-15T13:28:09.598 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:28:09.708 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is off 2024-05-15T13:28:09.808 INFO:teuthology.orchestra.console.smithi005:Power off completed 2024-05-15T13:28:09.914 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} duration: 4511.460448265076 failure_reason: 'Command failed (workunit test erasure-code/test-erasure-eio.sh) on smithi005 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae2481253f9b2fa3451954e615eedb67ebdd5314 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh''' flavor: default owner: scheduled_rzarzynski@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c43d76d00002482d89e44aff3b00cd8d status: fail success: false 2024-05-15T13:28:09.915 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-15T13:28:10.004 INFO:teuthology.run:FAIL